python - Optimizing memory managemente for huge variables in Jython -


I'm using approximately 7 to 10 MB as CSV file input, around 65,000 lines in which many The data fields will be hosted: for each of them, its acetid, FQDN and six property fields (see below for an example) 

  LCEEMV.V.Pray Cropper | SCCC_0002001 Prefusion | SVC_0001086 | Preframement SVCB00008660 | | Infrastructure Management LSER_0053150 | Wastecity-ADMGGPRACORP SVCC_0002001 | Prefudence | SVC_0001086 | Preproduction SVCB_0000160 | Infrastructure Management

A csv.reader object repeated on each line to use the I and its contents to save RAM: 

 < Code> prop_ COLUMNS: prop_value = row [column] data for prop, column in [[(FQDN, prop)]. Add (prop_value)

Then, when the input file is completely copied, it's time to dump it into an output file using a different syntax: 

  FQDN, property_name, property_values ​​

Th works flawlessly for smaller CSV files, however, it is currently sometimes host memory allocated huge CSV, JVM Stops based on usage, in which it is running. This is such a data.add bit which I believe is responsible for it.

Note that the input file is not necessarily sorted by FQDN, so I can not just line up the line and when the output is copied to the input, a second entry is found with the same FQDN They should be merged.

I want to map this var to a temporary file and use it before, but I am also not sure that it is possible and / or easy to implement. Now I am scared

It is not possible to use a database

I am still not familiar with everything that presents me with Python, so the ideal solution can be right in front of me and I see it I am not able to ... I hope A skilled person with someone can help a fellow developer

You do not have a full dataset stored in memory If you can, you can view using disk (temporary) as storage, as each FQDN can be saved in one file.

There are some things (it's half Cdmokod!): 

  temp_dir = tempfile.mktmpdir () Get: # FQDN and property line in CSV fqdn_filepath = os .path.join (temp_dir, FQDN.name) if not os.path.exists (fqdn_filepath): # Create another it from scratch: # Add the existing properties in the current FQDN file # has now ended redintegrate CSV You have a bunch of FQDN files. Now output as one-by-one with Opan_autput_fail () processed by each one was sorted fqdn_file (os.path.files_in_dir (temp_dir)): output.write_fqdn_data () Delete temp_dir

-

Comments

Post a Comment

Popular posts from this blog

sqlite3 - UPDATE a table from the SELECT of another one -

I have the first table article where some data is missing. In the table, articles_tmp I have a part of the missing data. So I want to update the articles. Where the path path is data in articles_tmp.path where the path is not zero . With SQLite , it does not work: update article SET path = (SELECT at.path FROM Articles_tmp as AS, WHERE at.article_id = a.id and a.asp, the text is in the form and not on. It has been transmitted) where the path is null MS Access Along with, it works fine: Update Articles Join the Inner Articles at articles_tmp articles.id = articles_tmp.id SETP articles.path = articles_tmp.path Can anyone help? Sin CE SQL Server Syntax, which I provided, was not working, here this solution With whom I have come. A little modification of your original query, but with my "correlation" suggestion added. There is a SQL Bella for this: Here is the code: update article SET path = (SELECT path from articles_tmp AS T WHERE t.id
Read more

c# - Showing a SelectedItem's Property -

So I have enabled a rightclick option for my DataGrid. I want to display the only chosen property but it is not behaving like what I would like it to display my name and additional. Public category Paymentinfo {public int PaymentNo {get; Set; } Public string date {get; Set; } Public Double Pay {get; Set; } Public Double Principle {get; Set; } Public Double Interest {get; Set; } Public Double Balance {Received; Set; }} Private Zero MenuItem_OnClick (Object Sender, RoutedEventArgs e) {MessageBox.Show (AmortGrid.SelectedItem.ToString ()); } I'm trying to apply it without using a View Module! If I place a breakpoint where the messagebox and the recipient place it on selected ITEM, then this property will display the payment. A date-payment-principle-interest-balance is the only value that needs payment. Was hoping that it would be something like this MessageBox.Show (AmortGrid.SelectedItem.PaymentNo.ToString ()); When you call ToString () So you get the class typ
Read more

javascript - Render HTML after each iteration in loop -

I'm trying to gradually increase the font size of text on a web page. The code works in my place, although the new HTML / CSS does not render after every iteration of the loop and when all this is done, then display only the text of 100 px size. To see the text as if it is slowly zooming, I need to do this JavaScript is down because it is being used from a different file here is what I have ... < Pre> & lt; P class = "game-title" style = "font-size: 50px" & gt; Test & lt; / P & gt; Function sleep (milliseconds) {var start = new date (). GetTime (); For (var i = 0; i & lt; 1e7; i ++) {if ((new date). GetTime () - start) & gt; Milliseconds {break; CSS ('font-size', 'parseInt ($ (' game-title '). CSS (' font-size ')) + 1 + "pixels"); } While (parasont ($ ('# text'). Css ('font-size')) & lt; = 100) {sleep (1000); IncreaseSize (); Using CSS: Use of jQuery (sample):
Read more