Why is there a limit on the number of counters allowed in a Hadoop map reduce job?

I am using Hadoop map-reduce and got a CountersExceededException when I tried to programatically create many job counters. I know that I can increase the number of counters that are allowed through the configuration files, but does anyone know:

a) why there is a limit on map-reduce counters

and

b) if it is a good/bad idea to increase the max number of map-reduce counters?

I am using hadoop 0.20.2

Answers


See the section on Counters in this post.

Counters

Counters represent global counters, defined either by the Map/Reduce framework or applications. Applications can define arbitrary Counters and update them in the map and/or reduce methods. These counters are then globally aggregated by the framework.

Counters are appropriate for tracking few, important, global bits of information. They are definitely not meant to aggregate very fine-grained statistics of applications. Counters are very expensive since the JobTracker has to maintain every counter of every map/reduce task for the entire duration of the application.


Just in case you want to up the maximum number of counters.

Last I saw the code, the maximum number of counters has been hard coded in the code and not configurable. Please check the blog entry I have written some time back for more details and also go through the latest code to check if this has been modified to make it configurable.


Need Your Help

Linux fakeroot confusion

linux linux-kernel sudo fakeroot

I have lot of confusion over fakeroot utility. I am trying to add a character node to /dev folder to two rootfs. Owner for rootfs_1 is myself whereas owner of second is root.

Get synonyms from synset returns error - Python

python nlp nltk wordnet

I'm trying to get synonyms of a given word using Wordnet. The problem is that despite I'm doing the same as is written here: here, it returns error.