java - limit of ArrayList<String>

I am trying to read a text file and store every line into ArrayList However, the text file is too long (about 2,000,000) lines and error: java.lang.OutOfMemoryError occurs.

How do i know if the arraylist is full and then create another arraylist to store the remaining data automatically?

Sorry for my poor english.

Thanks for your help.

Answers


2 million lines is far beyond the maximum size for Java Collection (INTEGER.MAX_VALUE or 2 billion indexes).

You are more likely to have heap space outOfMemory error. You can do either

  1. Increase your JVM maximum heap memory allocation.

java -Xmx4g

4g = 4GB.

The default maximum heap size is half of the physical memory up to a physical memory size of 192 megabytes and otherwise one fourth of the physical memory up to a physical memory size of 1 gigabyte.

http://www.oracle.com/technetwork/java/javase/6u18-142093.html

  1. as konsolas recommends, read line by line and store it into a file and flush the variable.

Hope it helps!


This depends on what you are planning to do with the file. You're definitely not going to be able to store all of it in memory, as shown by your error.

Depending on what you're trying to do with the file, processing it in blocks and then saving it would be a better idea. For example:

  • Read the first 1000 lines of the file
  • Process these lines/save into another file, etc.
  • Read the next 1000 lines
  • etc.

An ArrayList can theoretically hold 2,147,483,647 items. (max int)


I don't think ArrayList has a limit. It can grow bigger and bigger until the memory runs out (or the size reaches Integer.MAX_VALUE or somewhere like that). So in theory, if you have infinite memory, you can store infinite stuff in ArrayList.

Solution 1: Plain Brute Force (NOT RECOMMENDED)

You surround the reading file part with try..catch. When it catches an OutOfMemoryError, you know that that is the limit. So this is how you know what the array list's limit is. Actually, it is not the array list's limit. It's your PC's limit! So creating a new array list is no use...

This is not recommended because an OutOfMemoryError should never be caught. It tells you that there must be something wrong with your design.

Solution 2: A softer way

You read part of the file at a time. You can first read, say, 1/3 of the file and store it in an array list. Then you do some processing with it. After that, you read the next 1/3 of the file and store it in the same array list and overwrite the old lines and process it.

Please note that you should call clear() when you are done with a portion of the file. This way, the very helpful Garbage Collector will collect the processed information (1/3 of the file) for you and free up some memory!


As the other answers suggested, your problem is because you run out of memory before your ArrayList is full. If you still don't have enough memory after increasing the heap space size, BigArrayList will solve your problems. It functions like a normal ArrayList and automatically handles swapping data between disk and memory. Note that the library currently supports a limited number of operations, which may or may not be sufficient for you.


Need Your Help

Prevent status bar from auto-hide in full screen activity

android overlay android-fullscreen android-statusbar

I want status bar to be transparent and overlay on my activity while navigation bar is displayed normally.

Best options for image resizing

c# image-processing

I have resize images exceeding a max size. Methods I tried so far are not good enough :-(