troubleshooting

Halfway through the week I realised that I might have overreached myself just a little by promising to deliver the future and then most of the fun I had this week was still to come.

So, I'll postphone the future to next week and give you a short troubleshooting tip this week. For sometimes things go wrong. Networks can flap, disks can fail, processors can overheat. And then there's the OS, the JVM, ... Sometimes it is just amazing that anything works at all.

There is of course the Status tab in the HTTPBackend Fulcrum that has live graphs to study, but there is also the Requests and Threads page in the Developer tab.

Now this is a static page and I needed the information on just one specific thread at a more frequent interval. So I wrote a new tool :

<mapper>
  <config>
    <endpoint>
      <grammar>res:/tools/kerneldetail/
        <group name="threadname">
          <regex type="anything"/>
        </group>
      </grammar>
      <request>
        <identifier>active:xslt</identifier>
        <argument name="operand">netkernel:/k</argument>
        <argument name="operator">res:/resources/stylesheets/kerneldetail.xsl</argument>
        <argument name="threadname" method="as-string">arg:threadname</argument>
      </request>
    </endpoint>
  </config>
  <space>
    <fileset>
      <regex>res:/resources/stylesheets/.*</regex>
    </fileset>
    <import>
      <uri>urn:org:netkernel:xml:core</uri>
      <private/>
    </import>
    <import>
      <uri>urn:org:netkernel:ext:system</uri>
      <private/>
    </import>
  </space>
</mapper>

Did I say write ? No code (after no sql it is now time for the new paradigm, no code) of course. The netkernel:/k resource provides the raw data and I filter that with xslt :
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:nk="http://netkernel.org" 
                version="1.0">
  <xsl:output method="xml" 
              indent="yes" 
              encoding="UTF-8" 
              omit-xml-declaration="yes"/>
  <xsl:param name="threadname" nk:class="java.lang.String" />


  <xsl:template match="/k">
    <xsl:copy-of select="threads/thread[name=$threadname]"/>
  </xsl:template>
</xsl:stylesheet>

And that's it. Put in a small script ... for example like this :
#!/bin/sh
while true
do
 curl http://localhost:1060/tools/kerneldetail/ConcurrentCacheCullThread >> /var/tmp/culler.out
 sleep 10
done
And collect all the data you want with minimum system impact.

So, next week I will deliver the future as promised.

space listing

I remember being disappointed when I learned that an asteroid belt is nothing like what I knew from videogames (also an excellent song by Lana Del Rey), where you have to maneuver your ship between rocks with hardly any space between them. The truth is that you can pass through an asteroid belt without even seeing or coming close to any asteroid at all!

Space is like that. Contains a near infinite number of things but if you don't know where to look it might as well be empty.

Spaces in ROC (because of the way I set up your mind you might read that as "rocks in space") can also be like that. Sometimes you just want to know what is there before you go in and request it.

Note that this is not a logical train of thought. Because - and I quote Peter Rodgers - how do you enumerate a potentially infinite set?

It is however a practical train of thought, one that can only be followed through for spaces that are enumerable (whatever that means for the given space type). Such as a fileset. And of course, there's a tool in NetKernel that allows you to do that. Try the following in the Scripting Playpen :

<sequence>

  <request assignment="response">

    <identifier>active:sls</identifier>

    <argument name="space">urn:org:netkernel:ext:system</argument>

  </request>

</sequence>


Here's a snippet of the result :

<dir id="res:/etc/" name="etc">

  <res id="res:/etc/messages.properties" name="messages.properties"/>

  <dir id="res:/etc/system/" name="system">

    <res id="res:/etc/system/Books.xml" name="Books.xml"/>

    <res id="res:/etc/system/Docs.xml" name="Docs.xml"/>

  </dir>

</dir>


Now we are getting somewhere! This opens up possibilities for a resource oriented ftp server or a resource oriented content management system or a <your idea here>.

When you read the documentation for sls you'll notice that there are plans for some future enhancements. Come and see the future in next week's entry!

What did you expect ?

One of these days Peter Rodgers is bound to mention in the NetKernel Newsletter that I repeat/rehash a lot of what he and Tony have been saying all along. Of course I do. What did you expect ?

I want to focus on that question today. For in these days of multi-core systems, cheap memory (buy lots) even cheaper storage (buy lots lots) and unlimited bandwidth that question is often not even thought of until an application is going through the final performance tests.

And at that point your company's performance guy (been there, done that) might say - bad - things like :
- Those response times are - imo - not acceptable for production, what did you expect ?
- That is a very high throughput you've got there, what did you expect ?
- The cpus on the system are going red-hot, what did you expect ?

With a deadline looming like the sword of Damocles above your head, these are not things you want to hear.

The question "What do you expect ?" should be asked a lot earlier, it should be asked in the design phase, before any coding is done. The cost of overhauling a design is a lot lower than the cost of overhauling a completely finished application.

Now, next to the fact that NetKernel has an extremely light footprint on your systems as well as superior caching, it also has the tools to help you stay within the limits of what you expect.

Take nkperf. You can install this tool from the repository and it tells you how your system behaves. Not relevant you say ? Read this newsletter entry again. Or if you'd rather not have a rehashed ;-) story, look at what I found this morning on my own laptop (Windows, dual core intel) :

Oops, there is something very fishy with my caching performance. Especially when you look at this :

The second graph is from a virtual machine (CentOS, single core), running on the same laptop !

So why is the blue line flat on the first graph ? I don't know, but I better find out before I make any conclusions about applications running on that system. And that is the point of my story. You must know what you expect, what the constraints on your application are before you write it. And NetKernel can help you with that.


Footnote :
On seeing the above issue, the 1060 Research crew jumped in and pointed out the obvious cause of the flatliner. I've got a pretty speedy laptop. And to optimize caching, NetKernel has a cost threshold parameter described as : cost of an item must be at least this high before eligible for caching. Can you see where this is going ? The native machine (but not the vm) responded so fast that nothing was costly enough to be cached. Hence the flatliner.

Easily ammended, I set the cost threshold to 0. And look :

Five Step Tango

You know how you can often discover new things from re-reading (re-hearing) an old story ? I'm sitting in on a NetKernel training in the Twin Cities this week and while I have been in a similar training two or three times already, the following never took hold in my mind ...

Writing a physical endpoint contains five steps.

Now, such a statement is rather important given that there's nothing but spaces and endpoints (within those spaces) in NetKernel! So I don't know how I missed it in the previous sessions. Must be my selective hearing impediment.

Step 1. What is it you want ?
From the point of view of the physical endpoint that just got resolved to, this is rather important information. And it is available :
context.getThisRequest();

Step 2. What do I need to satisfy this request ?
Well, there are several possibilities here. Arguments passed to the request for example :
context.getThisRequest.getArgumentValue("...");
Also, subrequests may be needed :
context.createRequest(...);

Step 3. Add value
Be creative here!

Step 4. Prepare a response
Once the endpoint has done what it is supposed to do, it is time to create a response.
context.createResponseFrom(...);

Step 5. Set metadata on the response
This is an optional step (the five step tango also really has only four steps), most of the time the defaults will be fine. But you can :
response = context.createResponseFrom(...);
response.setMimeType("text/plain");

There, that is all there is to writing a physical endpoint. Enjoy!