Web Page Tools and Browser CGI flash Plugins and Dynamic Controls

Forking PHP!

One place where people seem to stumble with PHP is with long running PHP processes or parallel processing. The pcntl extension gives you the ability to fork PHP processes and run lots of children like many other unix daemons might. We use this for various things. Most notably, we use it run Gearman worker processes. While at the OReilly Open Sourc Convention in 2009, we were asked about how we pulled this off. So, we are releasing the two scripts that handle the forking and some instructions on how we use them.

This is not a detailed post about long running PHP scripts. ; Maybe I can get to the dos and don'ts of that another time. ; But, these are the scripts we use to manage long running processes. ; They work great for us on Linux. ; They will not run on Windows at all. ; We also never had any trouble running them on Mac OS X.

The first script, prefork.php, is for forking a given function from a given file and running n children that will execute that function. There can be a startup function that is run before any forking begins and a shutdown function to run when all the children have died.

The second script, prefork_class.php, uses a class with defined methods instead of relying on the command line for function names. This script has the added benefit of having functions that can be run just before each fork and after each fork. This allows the parent process to farm work out to each child by changing the variables that will be present when the child starts up. This is the script we use for managing our Gearman workers. We have a class that controls how many workers are started and what functions they provide. I may release a generic class that does that soon. Right now it is tied to our code library structure pretty tightly.

We have also included two examples. They are simple, but do work to show you how the scripts work.

You can download the code from the dealnews.com developers' page.

UPDATE: I have released a Gearman Worker Manager on Github.

Data Collection Protocols: AAA (RADIUS, Diameter, and TACACS+)

RADIUS, Diameter, and TACACS+ are three protocols for carrying Authentication, Authorization, and Accounting (AAA) information between a Network Access Server (NAS) that wants to authenticate its links or end users and a shared authentication server. The end user connects to the NAS, which in turn becomes a AAA client trying to authenticate the end user to the AAA server.

On this entry will concentrates on the last "A" of AAA (even though most of the time AAA is not primarily used for billing, but for authentication and authorization). However, when binding the accounting information with the authentication informs, the AAA protocols offer an interesting advantage for billing: the authenticated username.

RADIUS

Remote Authentication Dial-In User Service (RADIUS) is a client/server protocol developed by the IETF. The RADIUS client is typically a NAS, and the RADIUS server is usually a daemon process running on a UNIX or Windows server.

The RADIUS client (that is, the NAS) passes user information to designated RADIUS servers and acts on the returned response. RADIUS servers receive user connection requests via the NAS, authenticate the user, and then provide the NAS with configuration information necessary for it to deliver a specific service to the user.

Transactions between the RADIUS client and RADIUS server are authenticated with a shared secret key, which is never sent over the network. In addition, user passwords are sent encrypted between the RADIUS client and RADIUS server to eliminate the possibility that someone snooping on an insecure network could determine a user's password.

The accounting features of the RADIUS protocol can be used independently of RADIUS authentication or authorization. The NAS, which provides a service to the dial-in user (such as PPP or Telnet) is responsible for passing user accounting information to a designated RADIUS accounting server. At the start of service delivery, the NAS generates an "Accounting Start" packet describing the type of service being delivered and the user accessing the service. This packet is sent to the RADIUS accounting server, which returns an acknowledgment (the Accounting-Response) to the NAS, acknowledging that the "Accounting Start" packet has been received. At the end of the service delivery, the NAS client generates an "Accounting Stop" packet, describing the type of service that was delivered and session statistics such as elapsed time, input and output octets, and input and output packets. Here is the complete list of RADIUS accounting attributes, as described in RFC 2866: Acct-Status-Type, Acct-Delay-Time, Acct-Input-Octets, Acct-Output-Octets, Acct-Session-Id, Acct-Authentic, Acct-Session-Time, Acct-Input-Packets, Acct-Output-Packets, Acct-Terminate-Cause, Acct-Multi-Session-Id, and Acct-Link-Count. If the RADIUS server returns no response to the RADIUS client within a defined timeout, the request is resent a number of times. The RADIUS client can also forward requests to an alternate RADIUS server or servers in case the primary server is down or unreachable.

The UDP transport is a major issue in RADIUS accounting, where packet loss may translate directly into revenue loss.

For further references on RADIUS, refer to Table 3-9, which mainly focuses on the RADIUS accounting references. Note, for completeness, that other RADIUS RFCs are available: 2548, 2618, 2619, 2809, 2882, 3162, 3575, 3576, 3579, and 3580.

Note that the IETF RADIUS Extensions Working Group currently is focusing on extensions to the RADIUS protocol required to enable its use in applications such as IP telephony and local-area network AAA. To keep backward compatibility, the working group decided not to define new transports (such as TCP and SCTP).

Diameter

The Diameter protocol, standardized by the IETF Authentication, Authorization and Accounting working group, is the successor to the RADIUS protocol and was developed to overcome several limitations of RADIUS.

AAA protocols such as TACACS+ and RADIUS were initially deployed to provide dialup Point-to-Point Protocol (PPP) and terminal server access. Over time, with the growth of the Internet and the introduction of new access technologies, including wireless, DSL, Mobile IP, and Ethernet, routers and network access servers (NAS) have increased in complexity and density, putting new demands on AAA protocols.

Diameter introduced a couple of improvements compared to RADIUS:

For completeness, RFC 3589 also deals with Diameter, but not with accounting. Note also that several Diameter IETF drafts currently are in progress.

Tracking users from post processing the access.log

Most log analysers do not produce all statistics that a site might want to track. One of the important usage statistics that needs to be calculated is to track the number of active users (i.e. users that are actively using the system - not idle users) to try and correlate performance events with level of usage on the system.
It is possible to post process the access.log files using a set of UNIX script to summarize the information as the basic information such as userid and date time are contained in the file. The key to processing this data is to:

• Filter out the superfluous data and isolate the userid, date and time from the access.log records. Remember a record is written to the access.log when the user is actively seeking a resource on the web server. This is a definition of an "active" transaction record.


• Summarize that subset by the desired time frame (say each minute of the day). This is to cut down the information to the basics that are needed to for analysis. After the summarization you should end up with a unique record for each user in that minute of the day for the dates you require.

Note: The code below is not optimized and is provided as is. I am not a great script programmer and I am sure some of you out there would improve on it.

The script below is an example of such a set of commands to summarize that information:

#!/usr/bin/ksh
# Find all the access.log files in the log directory
# 1) We read each record from the access.log files (cat)
# 2) We filter out the console (grep)
# 3) We get the date time and userid from the access log record (cut)
# 4) We substring the date/time to chop off the seconds, the first "[" char and move the fields around (awk)
# 5) We filter out any records containing "-" (grep)
# 6) We isolate duplicates within that minute (sort)
# 7) We comma seperate the date/time and userid fields (sed)
# 8) Change the ":" within the date/time field to seperate date and time (sed) - Only the first one which should be at the end of the date
# At the end of pass 1 you should have a CSV that has a unique user entry for a date and time to the minute level. This implies that the user was active for that minute on that date.
ls -1 $SPLEBASE/logs/system/*access.log | while read _file
do
echo `date` : Started Parsing ${_file}
echo `date` : Pass 1 Started - Creating user list by minute
export _outfile=${_file}_userlist.csv
cat ${_file} | grep -v console |cut -f3,4 -d " " | awk '{print substr($2,2,17),$1}' | grep -v "-" | sort -u | sed -e "s% %,%g" | sed -e "s%:%,%" >${_outfile}
echo `date` : Pass 1 Ended - Check ${_outfile} for results
echo `date` : Lines created: `wc -l <${_outfile}`
done
echo `date` : Parsing finished

The above set of commands creates a file containing the date, time (to minute only) and userid and is comma delimited.
Note: Additionally the above information with coupled with information from SC_USER table will yield the Name of the user. It can also be used to determine who did not logon on (as you have the users who were active, the missing users are the ones that DID NOT login in the collection period).
This information may be loaded into data or post processed by another script to count the number of users in each minute of the day. The script below is such a script, which takes the files produced by the last process and creates active user counts:


#!/usr/bin/ksh
# Find all the *_userlist.csv files in the current directory
# 1) Sort the file to remove duplicates (just in case the files were combined)
# Otherwise you might count a user twice.
# 2) For each date/time combination work out how many users are active.
# 3) Output a date, time, number CSV file called _summary.csv
ls -1 *userlist.csv | while read _file
do
echo `date` : Started Parsing ${_file}
export _tmpfile=$$.tmp
cat ${_file} | cut -f1,2 -d "," | sort -u >${_tmpfile}
echo `date` : Pass 2 Started - Creating user summary
# Set the file name for the output
export _outfile=`basename ${_file} _userlist.csv`
export _outfile=${_outfile}_summary.csv
# Delete the file if it exists as you are going to append to it
rm -rf ${_outfile} 2>/dev/null
# Set the initial key values
export _key=""
export _count=0
# Loop through file and count number of instances
cat ${_tmpfile} | sort -u | while read _key
do
export _numrows=`cat ${_file} | grep "${_key}" | wc -l`
echo ${_key},${_numrows} >>${_outfile}
done
rm -rf ${_tmpfile} 2>/dev/null
echo `date` : Pass 2 Ended - Check ${_outfile} for results
echo `date` : Lines created: `wc -l <${_outfile}`
done
echo `date` : Parsing finished

This will produce a file that contains date, time (to the minute) and count of active users in a comma delimited format that can be loaded into Excel to produce relevant graphs.

Oracle Gateway Master Note

Master Note for Oracle Gateway Products

1. Concepts and Availability

2. Downloading Gateway Products

1. Oracle Technology Network - OTN -

2. Oracle eDelivery

3. Installation and Configuration

4. Troubleshooting

Web Design - Why is it Important to Me?