python - Optimizing memory managemente for huge variables in Jython -


I'm using approximately 7 to 10 MB as CSV file input, around 65,000 lines in which many The data fields will be hosted: for each of them, its acetid, FQDN and six property fields (see below for an example) 

  LCEEMV.V.Pray Cropper | SCCC_0002001 Prefusion | SVC_0001086 | Preframement SVCB00008660 | | Infrastructure Management LSER_0053150 | Wastecity-ADMGGPRACORP SVCC_0002001 | Prefudence | SVC_0001086 | Preproduction SVCB_0000160 | Infrastructure Management

A csv.reader object repeated on each line to use the I and its contents to save RAM: 

 < Code> prop_ COLUMNS: prop_value = row [column] data for prop, column in [[(FQDN, prop)]. Add (prop_value)

Then, when the input file is completely copied, it's time to dump it into an output file using a different syntax: 

  FQDN, property_name, property_values ​​

Th works flawlessly for smaller CSV files, however, it is currently sometimes host memory allocated huge CSV, JVM Stops based on usage, in which it is running. This is such a data.add bit which I believe is responsible for it.

Note that the input file is not necessarily sorted by FQDN, so I can not just line up the line and when the output is copied to the input, a second entry is found with the same FQDN They should be merged.

I want to map this var to a temporary file and use it before, but I am also not sure that it is possible and / or easy to implement. Now I am scared

It is not possible to use a database

I am still not familiar with everything that presents me with Python, so the ideal solution can be right in front of me and I see it I am not able to ... I hope A skilled person with someone can help a fellow developer

You do not have a full dataset stored in memory If you can, you can view using disk (temporary) as storage, as each FQDN can be saved in one file.

There are some things (it's half Cdmokod!): 

  temp_dir = tempfile.mktmpdir () Get: # FQDN and property line in CSV fqdn_filepath = os .path.join (temp_dir, FQDN.name) if not os.path.exists (fqdn_filepath): # Create another it from scratch: # Add the existing properties in the current FQDN file # has now ended redintegrate CSV You have a bunch of FQDN files. Now output as one-by-one with Opan_autput_fail () processed by each one was sorted fqdn_file (os.path.files_in_dir (temp_dir)): output.write_fqdn_data () Delete temp_dir

-

Comments

Post a Comment

Popular posts from this blog

Member with no value in F# -

After I am trying to convert F # to the following C # code: [StructLayout (LayoutKind.Sequential, charset = CharSet.Ansi)] Private class DOCINFOA {[MarshalAs (UnmanagedType.LPStr)] public string nomeDocumento; [Marshall (UnmanagedType.LPStr)] Public string arquivoSaida; [Marshall (UnmanagedType.LPStr)] Public String Tipodo; } I am doing the following: Named place printer module RawPrinterHelper = [& LT; StructLayout (LayoutKind.Sequential, charset = CharSet.Ansi) & gt;] Type DOCINFOA = [& LT; MarshalAs (UnmanagedType.LPStr) & gt;] Member this.nomeDocumento [& LT; MarshalAs (UnmanagedType.LPStr) & gt;] Member this.arquivoSaida [& LT; MarshalAs (UnmanagedType.LPStr) & gt;] Member This Tipoda [/ code> But I get: One type of definition is necessary for one or more members . Can I vacate these members? you val code> member : [& lt; StructLayout (LayoutKind.Sequential, charset = CharSet.Ansi) & gt;] Type DOCINFOA = [...
Read more

java - Joda Time Interval Not returning what I expect -

मेरे पास निम्न प्रोग्राम है import java.util। *; Import java.text। *; आयात करें org.joda.time। *; सार्वजनिक श्रेणी के स्कोप कंट्रोल {सार्वजनिक स्थिर अंतराल मिलनसार () {दिनांक समय currDate = नया दिनांक समय (2008, 4, 4, 15, 30, 0, 0); दिनांक समय epochDate = नया दिनांकटाइम (2000, 1, 1, 12, 0, 0, 0); अंतराल अंतराल = नया अंतराल (युरोप डेट, करोडेट); वापसी अंतराल; } सार्वजनिक स्थिर शून्य मुख्य (स्ट्रिंग [] आर्ग्स) {डबल दिनबात = मिलते समय ()। ToDurationMillis () / 1000/60/60/24; StdOut.println (daysBtween); }} मुझे आउटपुट मिल रहा है: 3016.0 लेकिन जो मैं देख रहा हूं वह है: 3016.1458333333 मैं क्या कर रहा हूँ ? toDurationMillis एक लंबा लौटा देता है, और प्रत्येक प्रभाग को int के रूप में घोषित किया जाता है। जावा इस प्रकार इनट्स को लॉन्ग में परिवर्तित कर देगा और डिवीजन को लंबे समय तक लौटाना होगा। परिणाम को अंत में दोहरे रूप में परिवर्तित करना। डबल्स का उपयोग करके अभिव्यक्ति करने के लिए जावा को बताने के लिए, अभिव्यक्ति के किसी भी घटक को दोहरे रूप में घोषित करें। उदाहरण के लिए: ...
Read more

c# - Showing a SelectedItem's Property -

So I have enabled a rightclick option for my DataGrid. I want to display the only chosen property but it is not behaving like what I would like it to display my name and additional. Public category Paymentinfo {public int PaymentNo {get; Set; } Public string date {get; Set; } Public Double Pay {get; Set; } Public Double Principle {get; Set; } Public Double Interest {get; Set; } Public Double Balance {Received; Set; }} Private Zero MenuItem_OnClick (Object Sender, RoutedEventArgs e) {MessageBox.Show (AmortGrid.SelectedItem.ToString ()); } I'm trying to apply it without using a View Module! If I place a breakpoint where the messagebox and the recipient place it on selected ITEM, then this property will display the payment. A date-payment-principle-interest-balance is the only value that needs payment. Was hoping that it would be something like this MessageBox.Show (AmortGrid.SelectedItem.PaymentNo.ToString ()); When you call ToString () So you get the class typ...
Read more