Remove empty directories

This line removes all empty directories in the folders below.

find -depth -type d -empty -exec rmdir {} \;

NFS settings

Below are some NFS settings we have found performance optimal.

The mount points are in async mode and the NFS export has sync enabled.

* App server mount options in /etc/fstab:

_netdev,rw,async,soft,bg,intr,noatime

* NFS Server mount options for physical volume:

rw,suid,dev,exec,auto,nouser,async,noatime

This is the same as saying “defaults,noatime”.

* NFS Server export options (/etc/exports):

/nfs cs1.local(rw,sync) cs2.local(rw,sync) cs3.local(rw,sync)

remote shutdown linux

ssh -t user@192.168.178.101 “sudo shutdown -h now”

Contention in tomcat jdbc pool

Apache tomcat, at least in the 6.x version range, suffers from deadlock contention. This is a know issue. A new jdbc pool implementation has been provided by Filip Hanik. Here are the configuration documentation and the download link.

To enable the new pool you will need to add a resource to your tomcat context configuration, similar to

<Resource name=”csDataSource” auth=”Container” type=”javax.sql.DataSource”
factory=”org.apache.tomcat.jdbc.pool.DataSourceFactory”
maxActive=”100″ maxIdle=”10″
username=”sa”
password=””
testWhileIdle=”true”
testOnBorrow=”true”
testOnReturn=”false”
validationQuery=”SELECT 1″
validationInterval=”30000″
timeBetweenEvictionRunsMillis=”30000″
maxWait=”1000″
initialSize=”2″
removeAbandonedTimeout=”60″
removeAbandoned=”true”
logAbandoned=”true”
minEvictableIdleTimeMillis=”30000″
jmxEnabled=”true”
jdbcInterceptors=”org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer”
driverClassName=”org.hsqldb.jdbcDriver”
url=”jdbc:hsqldb:/data/FatWire/JSK/7.5Patch3/App_Server/apache-tomcat-6.0.18/default/data/hypersonic/csDB”/>

What to tune in ContentServer

In the process of deploying a Content Server website, various layers can be tuned. This blog gives an quick overview of what can be tuned.

Operating System

• tcp stack
• ulimit
• NFS

httpd

• workers
• keep alive
• timeouts
• compression

Servlet Engine

• workers pool size
• db pool size

Content Server

• rs cache
• page cache size
• various other configuration files
• pagelet configuration
• render:calltemplate style/pagecriteria
• whitespace (jsp coding and/or filter)

Satellite Server

• cache sizes

Java Virtual Machine

• heap size
• perm sizes
• garbage collector configuration

production ContentServer settings

Here are default settings for futuretense.ini that could be set on a production delivery server as a baseline for further tuning:

futuretense.ini

#resultset cache:

cc.cacheNoSync=false
cc.cacheResults=1500
cc.cacheResultsAbs=true
cc.cacheResultsTimeout=60

cc.AssetTypeCSz=1500
cc.AssetTypeTimeout=-1

cc.CategoryCSz=500
cc.ElementCatalogCSz=5000
cc.ElementCatalogTimeout=-1
cc.SiteCatalogCSz=5000
cc.SiteCatalogTimeout=-1
cc.StatusCodeCSz=10
cc.SystemACLCSz=25
cc.SystemACLTimeout=-1
cc.SystemInfoCSz=5000
cc.SystemInfoTimeout=-1
cc.SystemPageCacheCSz=10000
cc.SystemPageCacheTimeout=1440
cc.SystemUsersCSz=1000
cc.SystemUsersTimeout=-1

cs.batchtimeout=360000
# point eventhost to the local tomcat http port, not to apache or load balancer
cs.eventhost=http://localhost:8080

#page cache:
cs.freezeCache=no
cs.alwaysusedisk=no
cs.nocache=false
ft.filecheck=no

cs.pgCacheTimeout=0

#publishing/page cache management
cs.manage.expired.blob.inventory=false
# set recordBlobInventory=false if only MungoBlob is used
# BlobAttributes get a new id when the asset is saved.
# ie they expire by getting a new url and do not have to be managed by CacheServer
cs.recordBlobInventory=false
cs.mirrorrowsperpost=6
cs.mirrorthreads=2
ss.flushall=false
cs.expireonly=true

cs.sitepreview=delivery

#Misc:
cs.httpvariables=false
cs.session=true
#set a session timeout reasonable for you site and visitor behaviour
cs.timeout=9000
#make sure that this number if large enough so the db is not trashed for generating new ids.
cs.uniqueidpoolsize=10000
cs.wrapper=false

# make sure the ft.sync is not blank and that all cluster members have the same value.
ft.sync=mycluster
ft.usedisksync=<your path>

#security:
secure.CatalogManager=true
secure.DebugServer=true
secure.TreeManager=true
security.checkpagelets=true
cc.security=true
bs.security=false
bs.invalidheadernames=Set-Cookie

# make sure that the values for HTTP_HOST and PROTOCOL are blank.
cs.HTTP_HOST=
cs.HTTP_PROTOCOL=

futuretense_xcel.ini

# set xcelerate.approval.dependency.chunksize on the source
xcelerate.approval.dependency.chunksize=250

# point batchhost to one cluster member in your source/staging cluster.
xcelerate.batchhost=localhost:8080
# used on source
xcelerate.batchloadsizeonpublish=250
# used on target
xcelerate.batchsavesizeonpublish=250

xcelerate.donotregenerate=*
xcelerate.enableinsite=false

Various other settings that are relevant:

System Properties

Add -DenableErrorPropagation=true to you java startup line. This helps to detect exceptions thrown in called elements.

Ulimit open files

The number of open files per process is in general too low by default. You need to increase this number to a higher number: 64k seems to be a high enough number for most production environments.

on Solaris can be set on from /etc/system.
On OS X, it be set in /etc/sysctl.conf.
Under Linux, these settings are often in /etc/limits.conf.

Publish destination Configuration

Add PUBLISHASSETTYPES=false to you publish destination configuration. Now the assettypes and elements are not published on each publish. This gives a performance increase as well as a decrease in chances of corruption.
Also, the table FW_PubDataStore can be cleaned each day when no publish is runnning, directy with SQL. For CS7.5 and 7.6 the publishing process leave orphan rows in this table that can be removed.

SiteCatalog

To reduce the chances of surprises make sure that there are no blank values for sscacheinfo and cscacheinfo in SiteCatalog for you Templates and SiteEntries.

httpd

The settings for apache httpd are explained in the blob about

netstat stat script

small script to count netstat states
netstat -an|awk ‘/tcp/ {print $6}’|sort| uniq -c

Clone Content Server install

Below is a list of changes to make when you clone a Content Server installation from one box to another.

This post assumes that you have already copied the database schema, the Content Server folder, the web-application and the shared directory.

If you leave the file locations in place the operation is much simpler, and you would only need to change a handfull of host/port combinations. Those are marked with *.

For contentserver database tables
DEFDIR in SystemInfo
update SystemInfo set defdir = ‘/opt/fatwire-del/data’ + SUBSTRING(defdir, 20, LEN(defdir)) where defdir like ‘/opt/fatwire/shared/%’
HOST AND PORT in SystemSatellite*
DEST column in PubTarget*
WORKINGFOLDER column in SearchEngineMetaDataConfig

maybe PROPERTIES in AssetIndexSourceConfig
maybe PROPERTIES in AssetQueues

ini files:
assetframework.ini:afk.publishdata
assetframework.ini:afk.historydata
catalog.ini:mwb.cartsetdir
databaseloader.ini:dbl.qfile
databaseloader.ini:dbl.cdfile
futuretense.ini:cs.xmlfolder
futuretense.ini:cs.pgcachefolder
futuretense.ini:cs.pgexportfolder
futuretense.ini:cs.jsproot
futuretense.ini:cs.eventhost*
futuretense_xcel.ini:xcelerate.batchhost*
futuretense_xcel.ini:xcelerate.sePath
futuretense_xcel.ini:xcelerate.transformpath
futuretense_xcel.ini:xcelerate.workflowdir
futuretense_xcel.ini:xcelerate.objpubdir
futuretense_xcel.ini:xcelerate.defaultbase
futuretense_xcel.ini:xcelerate.base
futuretense_xcel.ini:transformer.imgdir
futuretense_xcel.ini:xcelerate.lockdir
futuretense_xcel.ini:cs.binarypath
futuretense_xcel.ini:xcelerate.pubkeydir
futuretense_xcel.ini:xcelerate.thumbnaildir
futuretense_xcel.ini:xcelerate.saveSearchdir
futuretense_xcel.ini:xcelerate.tempobjectsdir
gator.ini:mwb.searchdir
gator.ini:cc.urlattrpath
gator.ini:mwb.path
visitor.ini:vis.genclasspath
visitor.ini:vis.rulesetxmlpath
visitor.ini:vis.compileclasspath
visitor.ini:vis.urlpath
visitor.ini:vis.sessiondata
visitor.ini:vis.path
writequeue.ini:q.storagefilepath
dir.ini:jndi.baseURL*

web.xml

inipath
NEW PATH

WEB-INF/classes (also check these on remore satellite server)
commons-logging.properties:logging.file
satellite.properties:contentserver.installation.folder
satellite.properties:host*
satellite.properties:port*
SSOConfig.xml:casUrl (when using WEM)*
cs-cache.xml and ss-cache.xml for the port and hostnames after Patch5 is installed. *

probably also log4j.xml

maybe startup files in tomcat/bin
maybe files in tomcat/conf

ContentServer jvm settings

The JVM has many options to set. Below are a list of settings that I believe are a reasonable start for the 64-bit Oracle JVM, for ContentServer production systems with mulitple cores available:

-server
-verbose:gc
-Xloggc:$CATALINA_BASE/logs/gc.log
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintHeapAtGC
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp
-Xms2G
-Xmx2G
-XX:NewRatio=3
-XX:SurvivorRatio=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-Xss2m
-XX:MaxPermSize=512m
-XX:PermSize=256m
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-Djava.awt.headless=true
-Dfile.encoding=UTF-8

I advise to add the following 3 settings to enable JMX remote access:

-Dcom.sun.management.jmxremote.port=51115
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

The above will start the remote server with no password. See Monitoring and Management via JMX for more background on JMX.

A complete overview of JDK6 JVM options have a look at the jdk6options page.

The GC logs can be analyzed and graphed with Tagtraum’s GCViewer.

Performance follows availability

Challenges that one encounters when hosting a database in the Cloud.

InfoQ: Availability, the Cloud and Everything.

Seems that the individual ec2 cloud node is not that reliable yet.

Also, I derived performance follows availability from this presentation.