Ambari scaling memory for all services

Initially I had two machines to setup hadoop, spark, hbase, kafka, zookeeper, MR2. Each of those machines had 16GB of RAM. I used Apache Ambari to setup the two machines with the above mentioned services.

Now I have upgraded the RAM of each of those machines to 128GB.

How can I now tell Ambari to scale up all its services to make use of the additional memory?

Do I need to understand how the memory is configured for each of these services?

Is this part covered in Ambari documentation somewhere?

Answers


Ambari calculates recommended settings for memory usage of each service at install time. So a change in memory post install will not scale up. You would have to edit these settings manually for each service. In order to do that yes you would need an understanding of how memory should be configured for each service. I don't know of any Ambari documentation that recommends memory configuration values for each service. I would suggest one of the following routes:

1) Take a look at each services documentation (YARN, Oozie, Spark, etc.) and take a look at what they recommend for memory related parameter configurations.

2) Take a look at the Ambari code that calculates recommended values for these memory parameters and use those equations to come up with new values that account for your increased memory.


I used this https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/determine-hdp-memory-config.html

Also, Smartsense is must http://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.2.0/index.html

We need to define cores, memory, Disks and if we use Hbase or not then script will provide the memory settings for yarn and mapreduce.

root@ttsv-lab-vmdb-01 scripts]# python yarn-utils.py -c 8 -m 128 -d 3 -k True
     Using cores=8 memory=128GB disks=3 hbase=True
     Profile: cores=8 memory=81920MB reserved=48GB usableMem=80GB disks=3
     Num Container=6
     Container Ram=13312MB
     Used Ram=78GB
     Unused Ram=48GB
     yarn.scheduler.minimum-allocation-mb=13312
     yarn.scheduler.maximum-allocation-mb=79872
     yarn.nodemanager.resource.memory-mb=79872
     mapreduce.map.memory.mb=13312
     mapreduce.map.java.opts=-Xmx10649m
     mapreduce.reduce.memory.mb=13312
     mapreduce.reduce.java.opts=-Xmx10649m
     yarn.app.mapreduce.am.resource.mb=13312
     yarn.app.mapreduce.am.command-opts=-Xmx10649m
     mapreduce.task.io.sort.mb=5324

Apart from this, we have formulas there to do calculate it manually. I tried with this settings and it was working for me.


Need Your Help

Validate a form using model and no table

ruby-on-rails

How can I validate a form (such as a contact form) using a model that doesn't have a database table (so not standard record validation).

Production considerations for Spring Cloud, Spring Data Redis & Eureka

redis spring-cloud production spring-session spring-data-redis

I have a Spring Cloud micro services application spanning 4 server types: A security gateway, two UI servers and a REST APIs server. Each one of these will run on its own VM in a production environ...