Daemon documentation

Source code

ocspd.main

Initialise the ocspd module.

This file only contains some variables we need in the ocspd name space.

ocspd.FILE_EXTENSIONS_DEFAULT = 'crt,pem,cer'

The extensions the daemon will try to parse as certificate files

ocspd.DEFAULT_REFRESH_INTERVAL = 60

The default refresh interval for the ocspd.core.certfinder.CertFinderThread.

ocspd.MAX_RESTART_THREADS = 3

How many times should we restart threads that crashed.

ocspd.LOG_DIR = '/var/log/ocspd/'

Directory where logs and traces will be saved.

ocspd.DEFAULT_CONFIG_FILE_LOCATIONS = ['~/.ocspd.conf', '/etc/ocspd/ocspd.conf']

Default locations to look for config files in order of importance.

ocspd.core.daemon

This module bootstraps the ocspd process by starting threads for:

• 1x ocspd.scheduling.SchedulerThread

  Can be used to create action queues that where tasks can be added that are either added to the action queue immediately or at a set time in the future.

• 1x ocspd.core.certfinder.CertFinderThread

  • Finds certificate files in the specified directories at regular intervals.
  • Removes deleted certificates from the context cache in ocspd.core.daemon.run.models.
  • Add the found certificate to the the parse action queue of the scheduler for parsing the certificate file.

• 1x ocspd.core.certparser.CertParserThread

  • Parses certificates and caches parsed certificates in ocspd.core.daemon.run.models.
  • Add the parsed certificate to the the renew action queue of the scheduler for requesting or renewing the OCSP staple.

• 2x (or more depending on the -t CLI argument) ocspd.core.ocsprenewer.OCSPRenewerThread

  • Gets tasks from the scheduler in self.scheduler which is a ocspd.scheduling.Scheduler object passed by this module.

  • For each task:

    • Validates the certificate chains.
    • Renews the OCSP staples.
    • Validates the certificate chains again but this time including the OCSP staple.
    • Writes the OCSP staple to disk.
    • Schedules a renewal at a configurable time before the expiration of the OCSP staple.

  The main reason for spawning multiple threads for this is that the OCSP request is a blocking action that also takes relatively long to complete. If any of these request stall for long, the entire daemon doesn’t stop working until it is no longer stalled.

• 1x ocspd.core.ocspadder.OCSPAdder (optional)

  Takes tasks haproxy-add from the scheduler and communicates OCSP staples updates to HAProxy through a HAProxy socket.

ocspd.core.taskcontext

This module defines an extended version of the general purpose scheduling.ScheduledTaskContext for use in the OCSP daemon.

class ocspd.core.taskcontext.OCSPTaskContext(task_name, model, sched_time=None, **attributes)

Adds the following functionality to the scheduling.ScheduledTaskContext:

• Keep track of the exception that occurred last, and how many times it occurred.
• Renames ScheduledTaskContext’s subject argument to model.

__init__(task_name, model, sched_time=None, **attributes)

Initialise a OCSPTaskContext with a task name, cert model, and optional scheduled time.

Parameters:

• task_name (str) – A task name corresponding to an existing queue in the scheduler.
• model (ocspd.core.certmodel.CertModel) – A certificate model.
• sched_time (datetime.datetime|int) – Absolute time (datetime.datetime object) or relative time in seconds (int) to execute the task or None for processing ASAP.
• attributes (kwargs) – Any data you want to assign to the context, avoid using names already defined in the context: scheduler, task_name, subject, model, sched_time, reschedule.

set_last_exception(exc)

Set the exception that occurred just now, this function will return the amount of times the same exception has occurred in a row.

Parameters:exc (Exception) – The last exception. Return int:Count of same exceptions in a row.

Todo

Make sure two similar exceptions are treated as identical, e.g. ignore attributes that will be different every time. https://code.greenhost.net/open/ocspd/issues/15

ocspd.core.certfinder

This module locates certificate files in the supplied directories and parses them. It then keeps track of the following:

• If cert is found for the first time (thus also when the daemon is started), the cert is added to the ocspd.core.certfinder.CertFinder.scheduler so the CertParserThread can parse the certificate. The file modification time is recorded so file changes can be detected.
• If a cert is found a second time, the modification time is compared to the recorded modification time. If it differs, if it differs, the file is added to the scheduler for parsing again, any scheduled actions for the old file are cancelled.
• When certificates are deleted from the directories, the entries are removed from the cache in ocspd.core.daemon.run.models. Any scheduled actions for deleted files are cancelled.

The cache of parsed files is volatile so every time the process is killed files need to be indexed again (thus files are considered “new”).

class ocspd.core.certfinder.CertFinderThread(*args, **kwargs)

This searches directories for certificate files. When found, models are created for the certificate files, which are wrapped in a ocspd.core.taskcontext.OCSPTaskContext which are then scheduled to be processed by the ocspd.core.certparser.CertParserThread ASAP.

Pass refresh_interval=None if you want to run it only once (e.g. for testing)

__init__(*args, **kwargs)

Initialise the thread with its parent threading.Thread and its arguments.

Parameters:

• models (dict) – A dict to maintain a model cache (required).
• directories (iter) – The directories to index (required).
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we add new parse tasks to. (required).
• refresh_interval (int) – The minimum amount of time (s) between search runs, defaults to 10 seconds. Set to None to run only once (optional).
• file_extensions (array) – An array containing the file extensions of file types to check for certificate content (optional).

run()

Start the certificate finder thread.

refresh()

Wraps up the internal CertFinder._update_cached_certs() and CertFinder._find_new_certs() functions.

Note

This method is automatically called by CertFinder.run()

_find_new_certs()

Locate new files, schedule them for parsing.

Raises:ocspd.core.exceptions.CertFileAccessError – When the certificate file can’t be accessed.

_del_model(filename)

Delete model from ocspd.core.daemon.run.models in a thread-safe manner, if another thread deleted it, we should ignore the KeyError making this function omnipotent.

Parameters:filename (str) – The filename of the model to forget about.

_update_cached_certs()

Loop through the list of files that were already found and check whether they were deleted or changed.

If a file was modified since it was last seen, the file is added to the scheduler to get the new certificate data parsed.

Deleted files are removed from the model cache in ocspd.core.daemon.run.models. Any scheduled tasks for the model’s task context are cancelled.

Raises:ocspd.core.exceptions.CertFileAccessError – When the certificate file can’t be accessed.

check_ignore(path)

Check if a file path matches any pattern in the ignore list.

Parameters:path (str) – Path to a file to match.

static compile_pattern(pattern)

Compile a glob pattern and return a compiled regex object.

Parameters:pattern (str) – Glob pattern.

ocspd.core.certparser

This module parses certificate in a queue so the data contained in the certificate can be used to request OCSP responses. After parsing a new ocspd.core.taskcontext.OCSPTaskContext is created for the ocspd.core.oscprenewe.OCSPRenewer which is then scheduled to be processed ASAP.

class ocspd.core.certparser.CertParserThread(*args, **kwargs)

This object makes sure certificate files are parsed, after which a task context is created for the ocspd.core.oscprenewer.OCSPRenewer which is scheduled to be executed ASAP.

__init__(*args, **kwargs)

Initialise the thread with its parent threading.Thread and its arguments.

Parameters:

• models (dict) – A dict to maintain a model cache (required).
• minimum_validity (int) – The amount of seconds the OCSP staple should be valid for before a renewal is scheduled (required).
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we can get parser tasks from and add renew tasks to. (required).
• no_recycle (bool) – Don’t recycle existing staples (default=False)

run()

Start the certificate parser thread.

parse_certificate(model)

Parse certificate files and check whether an existing OCSP staple that is still valid exists. If so, use it, if not request a new OCSP staple. If the staple is valid but not valid for longer than the minimum_validity, the staple is loaded but a new request is still scheduled.

ocspd.core.ocsprenewer

This module takes renew task contexts from the scheduler which contain certificate models that consist of parsed certificates. It then generates an OCSP request and sends it to the OCSP server(s) that is/are found in the certificate and saves both the request and the response in the model. It also generates a file containing the respone (the OCSP staple) and creates a new ocspd.core.taskcontext.OCSPTaskContext to schedule a renewal before the staple expires. Optionally creates a ocspd.core.taskcontext.OCSPTaskContext task context for the ocspd.core.oscpadder.OCSPAdder and schedules it to be run ASAP.

class ocspd.core.ocsprenewer.OCSPRenewerThread(*args, **kwargs)

This object requests OCSP responses for certificates, after which a new task context is created for the ocspd.core.oscprenewer.OCSPRenewer which is scheduled to be executed before the new staple expires. Optionally a task is created for the ocspd.core.oscpadder.OCSPAdder to tell HAProxy about the new staple.

__init__(*args, **kwargs)

Initialise the thread’s arguments and its parent threading.Thread.

Parameters:

• minimum_validity (int) – The amount of seconds the OCSP staple is still valid for, before starting to attempt to request a new OCSP staple (required).
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we can get tasks from and add new tasks to. (required).

run()

Start the renewer thread.

schedule_renew(model, sched_time=None)

Schedule to renew this certificate’s OCSP staple in sched_time seconds.

Parameters:

• context (ocspd.core.certmodel.CertModel) – CertModel instance None to calculate it automatically.
• shed_time (int) – Amount of seconds to wait for renewal or None to calculate it automatically.

Raises:

ValueError – If context.ocsp_staple.valid_until is None

ocspd.core.ocspadder

Module for adding OCSP Staples to a running HAProxy instance.

class ocspd.core.ocspadder.OCSPAdder(*args, **kwargs)

This class is used to add a OCSP staples to a running HAProxy instance by sending it over a socket. It runs a thread that keeps connections to sockets open for each of the supplied haproxy sockets. Code from collectd haproxy connection under the MIT license, was used for inspiration.

Tasks are taken from the ocspd.scheduling.SchedulerThread, as soon

as a task context is received, an OCSP response is read from the model within it, it is added to a HAProxy socket found in self.socks[<certificate directory>].

TASK_NAME = 'proxy-add'

The name of this task in the scheduler

OCSP_ADD = 'set ssl ocsp-response {}'

The haproxy socket command to add OCSP staples. Use string.format to add the base64 encoded OCSP staple

__init__(*args, **kwargs)

Initialise the thread with its parent threading.Thread and its arguments.

Parameters:

• socket_paths (dict) – A mapping from a directory (typically the directory containing TLS certificates) to a HAProxy socket that serves certificates from that directory. These sockets are used to communicate new OCSP staples to HAProxy, so it does not have to be restarted.
• scheduler (ocspd.scheduling.SchedulerThread) – The scheduler object where we can get “haproxy-adder” tasks from (required).

_open_socket(key, socket_path)

Opens a socket located at socket_path and saves it in self.socks[key]. Subsequently it asks for a prompt to keep the socket connection open, so several commands can be sent without having to close and re-open the socket.

Parameters:

• key – the identifier of the socket in self.socks
• socket_path (str) – A valid HAProxy socket path.

:raises :exc:ocspd.core.exceptions.SocketError: when the socket can not

be opened.

__del__()

Close the sockets on exit.

run()

The main loop: send any commands that enter the command queue

Raises:ValueError – if the command queue is empty.

add_staple(model)

Create and send the command that adds a base64 encoded OCSP staple to the HAProxy

Parameters:model – An object that has a binary string ocsp_staple in it and a filename filename.

send(socket_key, command)

Send the command through self.socks[socket_key] (using self.socket_paths)

Parameters:

• socket_key (str) – Identifying dictionary key of the socket. This is typically the directory HAProxy serves certificates from.
• command (str) – String with the HAProxy command. For a list of possible commands, see the haproxy documentation

:raises IOError if an error occurs and it’s not errno.EAGAIN or

errno.EINTR

ocspd.core.certmodel

This module defines the ocspd.core.certmodel.CertModel class which is used to keep track of certificates that are found by the ocspd.core.certfinder.CertFinderThread, then parsed by the ocspd.core.certparser.CertParserThread, an OCSP request is generated by the ocspd.core.ocsprenewer.OCSPRenewer, a response from an OCSP server is returned. All data generated and returned like the request and the response are stored in the context.

The following logic is contained within the context class:

• Parsing the certificate.
• Validating parsed certificates and their chains.
• Generating OCSP requests.
• Sending OCSP requests.
• Processing OCSP responses.
• Validating OCSP responses with the respective certificate and its chain.

class ocspd.core.certmodel.CertModel(filename)

Model for certificate files.

__init__(filename)

Initialise the CertModel model object, and read the certificate data from the passed filename.

Raises:ocspd.core.exceptions.CertFileAccessError – When the certificate file can’t be accessed.

parse_crt_file()

Parse certificate, wraps the _read_full_chain() and the _validate_cert() methods. Wicth extract the certificate (end_entity) and the chain intermediates*), and validates the certificate chain.

recycle_staple(minimum_validity)

Try to find an existing staple that is still valid for more than the minimum_validity period. If it is not valid for longer than the minimum_validity period, but still valid, add it to the context but still ask for a new one by returning False.

If anything goes wrong during this process, False is returned without any error handling, we can always try to get a new staple.

Return bool:False if a new staple should be requested, True if the current one is still valid for more than minimum_validity

renew_ocsp_staple()

Renew the OCSP staple, validate it and save it to the file path of the certificate file (certificate.pem.ocsp).

Note

This method handles a lot of exceptions, some of then are non-fatal and might lead to retries. When they are fatal, one of the exceptions documented below is raised. Exceptions are handled by the ocspd.core.excepthandler.ocsp_except_handle() context.

Note

There can be several OCSP URLs. When the first URL fails, the error handler will increase the url_index and schedule a new renewal until all URLS have been tried, then continues with retries from the first again.

Raises:

• RenewalRequirementMissing – A requirment for the renewal is missing.
• OCSPBadResponse – Response is empty, invalid or the status is not “good”.
• urllib.error.URLError – An OCSP url can’t be opened (Python3).
• urllib2.URLError – An OCSP url can’t be opened (Python2).

Raises:

urllib.error.URLError/urllib2.URLError - when a URL/HTTP error occurs

Raises:

socket.error - when a socket error occurs

Todo

Send merge request to ocspbuider, for setting the hostname in the headers while fetching OCSP records. If accepted the request library won’t be needed anymore.

_check_ocsp_response(ocsp_staple, url)

Check that the OCSP response says that the status is good. Also sets ocspd.core.certmodel.CertModel.ocsp_staple.valid_until.

Raises:OCSPBadResponse – If an empty response is received.

_read_full_chain()

Parses binary data in self.crt_data and parses the content. The server certificate a.k.a. end_entity is put in self.end_entity, anything else that has a CA extension is added to self.intermediates.

Note

At this point it is not clear yet which of the intermediates is the root and which are actual intermediates.

Raises:CertParsingError – If the certificate file can’t be read, it contains errors or parts of the chain are missing.

_validate_cert(ocsp_staple=None)

Validates the certificate and its chain, including the OCSP staple if there is one in self.ocsp_staple.

Parameters:ocsp_staple (asn1crypto.core.Sequence) – Binary ocsp staple data. Return array:Validated certificate chain. Raises:CertValidationError – If there is any problem with the certificate chain and/or the staple, e.g. certificate is revoked, chain is incomplete or invalid (i.e. wrong intermediate with server certificate), certificate is simply invalid, etc.

Note

At this point it becomes known what the role of the certiticates in the chain is. With the exception of the root, which is usually not kept with the intermediates and the certificate because ever client has its own copy of it.

__repr__()

We return the file name here because this way we can use it as a short-cut when we assign this object to something.

__str__()

Return a formatted string representation of the object containing: "<CertModel {}>".format("".join(self.filename)) so it’s clear it’s an object and which file it concerns.

__weakref__

list of weak references to the object (if defined)