Akom's Tech Ruminations

In CI, I am using minikube for local testing on my nodes.  In the name of repeatable builds, I completely rebuild minikube after each build (minikube delete, minikube start).   I am also using an internal Docker Private Registry (DPR or DTR).  Same methodology applies if you are using Google Container Registry or the Amazon equivalent.  In order to authenticate to DPR, I am using the registry-creds addon.  It is normally configured interactively:

$ minikube addons configure registry-creds

Do you want to enable Docker Registry? [y/n]: y

It then asks you some questions and expects user input.  

Automating

I want this automated, as it will be running several times an hour.  The addon config does not take any parameters, so I'm taking a different approach.  The configure step creates some credentials in k8s, and it is possible to transfer them from one cluster to another.  This is how I am doing it:

Continue reading "Automating minikube configuration to log in to a Docker Private Registy."

Let's assume that you want to do the following:

1. Host the contents of your S3 bucket via YOUR.DOMAIN.COM
2. Use CloudFront
3. Use a LetsEncrypt cert

"What's the problem?", you may ask. There are plenty of tutorials for this stuff. Not exactly. A CloudFront Distribution will not let you add a CNAME until you have an SSL cert, but you can't use certbot to auto-provision an SSL cert until you are hosting from your domain.

Solution

The easiest solution is to initially generate your SSL cert manually. You can then use something like certbot-s3front to auto-renew.

Details

Continue reading "Bootstrapping S3/CloudFront with LetsEncrypt"

I'm running PowerShell on Linux for the sake of using PowerCLI. An interesting thing happens: when pwsh is run interactively, in a terminal, the following works fine when PowerCLI is already installed:

pwsh -c 'Get-InstalledModule VMware.PowerCLI'

When puppet runs it - it doesn't find the module:

Get-Package: /opt/microsoft/powershell/7/Modules/PowerShellGet/PSModule.psm1:9445
Line |
9445 |          PackageManagement\Get-Package @PSBoundParameters | Microsoft. …
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | No match was found for the specified search criteria and
     | module names 'VMware.PowerCLI'.

Error: 'pwsh -c "if (-not(Get-InstalledModule VMware.PowerCLI)) { Exit 1 }"' returned 1 instead of one of [0]

After some experimentation, I narrowed it down to a single environment variable missing when puppet executes the command: HOME. Set HOME, and powershell will find modules. That gets us the following recipe:

  $my_user = 'some_user'

  exec { 'Install PowerCLI':
    path    => ['/bin', '/usr/bin'],
    command => 'pwsh -c \'Install-Module -Force:$true -Name VMware.PowerCLI\'',
    unless  => 'pwsh -c \'if (-not(Get-InstalledModule VMware.PowerCLI -ErrorAction silentlycontinue)) { Exit 1 }\'',
    user    => $my_user,
    environment => [ "HOME=/home/${my_user}"]
  }

I'm testing a product that needs to be compiled with JDK 8 but tested (sometimes) on JDK 11. This is now possible to do with maven surefire (although that took some effort). With gradle, I was doing it as follows, which is terrible, even if the path comes from configuration:

// The old way:
test {
    executable = '/some/hardcoded/path/to/java'
}
 

Now that Gradle 6.7+ has built-in toolchains support, it's trivial to configure the compile toolchain:

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(11)
    }
}
 

But what about tests? What if I want to test with a different JVM? It's a little more verbose:

test {
    JavaToolchainService javaToolchainService = project.getExtensions().getByType(JavaToolchainService.class)
    def launcher = javaToolchainService.launcherFor{
        languageVersion = JavaLanguageVersion.of(11)
    }
    javaLauncher = launcher
    environment 'JAVA_HOME', launcher.get().metadata.installationPath //if your tests care about JAVA_HOME
}
 

I've always set up autossh in /etc/rc.local, but with CentOS 8 that doesn't work well (things start too early, etc). Luckily, there is a nicer way using systemd templates. Essentially, all you have to do is create one symlink and one config file per instance of autossh.

Example:

Make a config file

/etc/autossh/mything.conf:

OPTIONS=-N  -M 20000 -R8888:1.2.3.4:8888 5.6.7.8

Make a symlink

ln -s /usr/lib/systemd/system/[email protected] /etc/systemd/system/[email protected]

Test

systemctl start [email protected]
journalctl -xe

But it's better to automate:

Puppet Recipe

define autossh(
    $service=$name,
    $args,
  ) {

    file {"/etc/systemd/system/autossh@${service}.service":
      ensure => link,
      target => '/usr/lib/systemd/system/[email protected]'
    } ->
    file {"/etc/autossh/${service}.conf":
      content => "OPTIONS=${args}"
    } ->
    service {"autossh@${service}":
      ensure => running,
      enable => true,
    }

}

# and example usage - random port forwarding
autossh{'mything': args => '-N  -M 20000 -R8888:1.2.3.4:8888 host1'}
autossh{'mything2': args => '-N  -M 20000 -R8888:1.2.3.4:8888 host2'}
 

My company's security department decided to assign Antivirus exclusion policies based on the value of the windows computer description. That means that I need to set computer descriptions ( net config server /srvcomment:"new description" ) to the same value on a whole bunch of windows machines. Doing that by hand is unappealing, and I already have puppet, so here is a simple solution:

MYCLASS/lib/facter/wincomputer_description.rb

MY_CLASS/manifests/init.pp

And when I say "All versions of Windows" I mean that I tested it on Server 2008, 2012 and 2016.

This was oddly hard to figure out, and most tutorials either apply to only one version of windows or to outdated tools. The best way to do this that I found was using Local Group Policy. Now, how to automate this? The puppet localgrouppolicy module didn't work at all when I tried it (and has not been updated since 2014). The proper way to do this is of course with a Domain-based Group Policy, but my machines are not members of a domain.

Fortunately, there is a new Microsoft tool called LGPO that allows for some degree of command-line control of the Local Group Policy. Download "LGPO.exe" here.

First, let's make a reusable policy text file that we can import on all machines:

1. take a vanilla Windows machine that hasn't had any Group Policy customization, and use lgpo.exe to export the policy: "lgpo.exe /v /parse /u c:\windows\system32\GroupPolicy\User\Registry.pol" (at least that was appropriate in my case). You should get more or less empty output.
2. Then use the Local Group Policy Editor to change "Do not move deleted files to the Recycle bin" (under User Configuration -> Administrative Tools -> All Settings) to "Enabled"
3. Repeat step 1. You should see this one setting that you changed in the output. Redirect output to a file, this will be our text file
4. You can test that importing this file will change the setting: "lgpo /r myfile.txt". (Change the setting back first, run this, then re-open the Local Group Policy Editor to see the change)

Now, we can set up puppet:

Continue reading "Disabling Windows Recycle Bin with Puppet on all versions of Windows"

Let's say that for the sake of speed, you are running two slow things in parallel, but you want one to wait for the other.

parallel one: {
    node {
        sh "sleep 15"
    }
}, two: {
    node {
        //slow part:
        sh "sleep 10"  

        // now do something that needs "one" to finish.  There is a good chance that this will run too soon...
        sh 'wget http://something' //for example
    }
}
 

The problem, obviously, is that you can't be sure that the wget will run after part one finishes. (Let's assume that part one creates the file).

Continue reading "Jenkins Pipeline: parallel and waitUntil, waiting until the other branch finishes"

This turned out to be fairly simple, with only one gotcha: do not follow the how-to's out there that tell you to use expect fork. The process doesn't technically fork. When I had that setting enabled, upstart commands would hang under very specific but repeatable conditions (if the process was killed externally).

So, here is my upstart conf file:

Continue reading "Running Jenkins Swarm client as a service via Upstart"

Why Because my Jenkins setup sometimes starts containers and forgets about them. Either it thinks it failed to start one, or the container itself has trouble starting. Either way, I'm left with containers that are running, trying to connect to Jenkins in vain, forever. The proper way to fix this is probably to have the containers timeout at some point, but that mechanism is broken.

Anyway, the fix I have is a true hack: find containers that have been up more than 2 days and kill them. None of our jobs should run for more than about a day, so this is a safe limit. Here is a bash script to do this:

Continue reading "Docker: Automatically remove containers that have been running too long"

After hours of "Why does it work locally but not in Jenkins", this error boils down to StrictHostKeyChecking... In other words, since the job runs as a user on a random slave, and this user most likely doesn't have a known hosts file with an entry for the target system, this fails rather cryptically. You think that the user's keys don't work, but that's not the problem.

The whole setup boils down to :

1. Install ssh-agent plugin
2. Configure credentials with a valid ssh key for your target
3. Enable ssh-agent with that credential entry in your job config
4. Add StrictHostKeyChecking=no to whatever ssh command you are using. Some examples:
   • GIT:

     export GIT_SSH_COMMAND="ssh -oStrictHostKeyChecking=no"

   • SSH:

     ssh -oStrictHostKeyChecking=no ....

currentBuild.@result = hudson.model.Result.UNSTABLE

Alternatively, to mark the build aborted and terminate immediately instead:

throw new InterruptedException()

Continue reading "Marking Jenkins build UNSTABLE from environment inject groovy script"

1. Convert hundreds (in our case) of Maven jobs to regular (freestyle) jobs. There appears to be no simple script to do this, so I wrote one here
2. Use maven toolchains to compile and test with a different JDK. Although converting jobs is a viable option that requires no additional setup, using toolchains may be preferred in some cases. For one thing, Jenkins offers the "Perform Release" button on maven jobs, something that is difficult to emulate in freestyle jobs.

mount -v /mountpoint                                                                                                               
mount.nfs: timeout set for Wed Dec  9 11:09:27 2015
mount.nfs: trying text-based options 'rsize=8192,wsize=8192,intr,hard,addr=X.X.X.X'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: requested NFS version or transport protocol is not supported