Integrating JBoss Admin iOS App with Apigee App Perf Monitoring Tool

Ever since I left JBoss/Red Hat last year to work on InstaOps, a start up that I co-founded, I have always wanted to stay connected not only with many life long friends that I have made at Red Hat but also with amazing technologies that are built over there. Along this line, I came across JBoss Admin iOS mobile app and wondered what if I could integrate this app with iOS SDK that we have built. The app is very well designed thanks to Christos (@cvasilak) and uses standard iOS frameworks so integration literally took less than 10 minutes. If I were a brand new user, the end to end integration would have probably taken 20 min most of which would have been time taken to sign up and download the SDK. ;-). Here is a screenshot of app running on Simulator:

You can see the code changes that I had to make to integrate our SDK on GitHub.  Changes are very minimal and in terms of new lines of code, it's literally 2 files (App Delegate header and impl) and less than 5 lines of code. Once I ran the app on Simulator, following are screenshots of what I could see in our dashboard:

• Automagic Log Capture

• Automagic Network Capture

The one network error you see on the chart is because I had not started AS7 before I started the app. 

If you like what you see and are into mobile App development, pls give it a shot.  It's free. Let me know your feedback (@prabhatjha). We currently have SDK for native Android and iOS apps.

What does it mean to be JBoss Data Grid ??

Red Hat recently announced beta release of JBoss Data Grid (JDG) , a supported product based on Infinispan. Infinispan was first announced in 2009 and its first GA release was made in February, 2010. So it took more than three years for us, folks at JBoss, to announce a supported beta release. In case you are wondering why it took so long, here are few things to consider. Besides introducing several handy features such as virtual nodes, grouping API, deadlock detection and optimization, improvements with transactions, we wanted to make sure that it

• can scale in a large cluster. Scalability is a very generic term and its meaning is highly contextual. For JDG, we narrowed down it to two : client scalability and data scalability.
• Client Scalabilty:  Given a  percentage of data in virtual heap, does JDG handle more client requests as more nodes are added to the cluster? We measure throughput, 99 percentile response  time under read heavy, write heavy scenarios to verify this assertion.
• Data Scalability: Given a fix number of clients, how much data can be filled in virtual heap while performance metrics such as throughput and 99 percentile response time are still under user defined values. This is again tested under read heavy, write heavy scenarios. 
• is elastic. Does the cluster scale out and scale in? As nodes are added and removed, how long does it take for cluster to stabilize? This stability is measured by comparing data distribution before and after nodes leave and join.
• is resilient. While cluster is in active use and node/s crash, what's the impact on throughput, response time? Is there any data inconsistency? How long does it take to stabilize?
• is stable. Tests are run for long time and monitored for CPU, memory usage, number of full GC, average GC time.

This matrix gets quite complicated if you add various configurations that are available in JDG such as three different end points (Hot Rod, memcached, REST), partially versus fully replicated modes aslo known as "DISTRIBUTED" and  "REPLICATED" mode respectively,  synchronous vs asynchronous replication, number of virtual nodes, READ heavy vs WRITE heavy, L1 cache enabled vs disabled, File Cache Store, JDBC cache store, and so on. 

I have only touched the performance and scalability aspects in details. There are various other functional verification which gets quite complicated as well if you think in terms of OS+JVM+File System + DB combinations we have to certify. 

I can proudly said that we have come a long way and come GA availability, our customers will have in their hands an extremely scalable and highly available data grid platform. And I should mention linear scalability as well!! Wait should not be too long. ;-)

Lessons Learned from Scalability Testing

Currently I work on JBoss Portal project and recently I have been working on its performance and scalability part. The outcome of my recent effort is published at this blog entry. This was a very prolonged testing which required lots of patience, working with several teams, learning many new different tools and concepts. But now it feels like every bit of it was worth it. Here are the lessons that I learned not in any particular order:

• Performance and scalability testing is not easy. Besides it being heavily technical I believe it's an art and you mature by experience.
• Set a goal first otherwise there is no end to this. Be very concrete: I want my application to handle x concurrent users with response time of less than y second/s and it should scale more than z%.
  
• If the application/system you are trying to improve the performance of and making more scalable is not developed by you, it's a must that people who developed are aware of scope of your work and helpful. I had good fortune of this being the case. I do not even want to imagine how it would have been otherwise.
• The best way to find bottlenecks is taking several thread dumps and taking very close look at them. I found it the most revealing and helpful than any profiler (commercial or open source). TDA is a good tool to analyze thread dump. I used three profilers: JBoss Profiler, JProfiler, SAP's Memory Analyzer.
• Regardless of what people say, open source load testing tools serve most purposes. I found Grinder to be very straightforward, not that hard to configure, and very vibrant community.
• For managing distributed deployment, use or extend a distributed framework. I used Smartfrog components to manage instances of JBoss AS, start up load generator, start up apache web server and load balancer otherwise I would have had to log into around 15 servers to start/stop/configure these components.
  
• Try to optimize most common code execution path. Optimizing code base which crawls but is also used rarely wont help you much.
• JVM's default garbage collection configurations are pretty good. I have toyed with several configurations and none of those seem to have any significant impact in my use case. My point is that you play with GC configurations when you have fixed other problems.
  
• JConsole that comes free with Sun JDK is pretty good tool to observe GC behavior, memory footprint and different logical memory spaces (eden, perm etc) of heap. Use that as a first tool for your profiling.
  
• It's a common knowledge that database is always a bottleneck. Hibernate's show_sql switch is a very good friend. Make sure that you do not see queries being executed more than they should. Tuning those bumped up performance quite a bit.
• The first thing you turn off after you have done basic optimization is logging. It has a big overhead simply because logging requires write to file and disk access.
• Don't use NFS for testing.
• If servers are part of cluster, make sure that they all are part of same subnet and secluded from any other network traffic.
• Find a compromise between loading everything in memory and loading everything on demand (lazy loading). The former may show a good performance in the beginning but crawl once memory fills up and lot of full GC starts happening. The latter will have higher DB utilization. These settings are dependent on nature of your application and nature of how the application will be used. There is no universal setting. Let me know if you find one.
• This blog is a WIP. I will add more bullet points as I think of them. :-)