Akom's Tech Ruminations

Various tech outbursts - code and solutions to practical problems

Automating minikube configuration to log in to a Docker Private Registy.

Posted by Admin • Tuesday, March 14. 2023 • Category: DevOps

In CI, I am using minikube for local testing on my nodes.  In the name of repeatable builds, I completely rebuild minikube after each build (minikube delete, minikube start).   I am also using an internal Docker Private Registry (DPR or DTR).  Same methodology applies if you are using Google Container Registry or the Amazon equivalent.  In order to authenticate to DPR, I am using the registry-creds addon.  It is normally configured interactively:

$ minikube addons configure registry-creds

Do you want to enable Docker Registry? [y/n]: y

It then asks you some questions and expects user input.  

Automating

I want this automated, as it will be running several times an hour.  The addon config does not take any parameters, so I'm taking a different approach.  The configure step creates some credentials in k8s, and it is possible to transfer them from one cluster to another.  This is how I am doing it:

Continue reading "Automating minikube configuration to log in to a Docker Private Registy."

Linux Bootstrapping S3/CloudFront with LetsEncrypt

Posted by Admin • Thursday, August 5. 2021 • Category: DevOps, Linux

Let's assume that you want to do the following:

  1. Host the contents of your S3 bucket via YOUR.DOMAIN.COM
  2. Use CloudFront
  3. Use a LetsEncrypt cert

"What's the problem?", you may ask. There are plenty of tutorials for this stuff. Not exactly. A CloudFront Distribution will not let you add a CNAME until you have an SSL cert, but you can't use certbot to auto-provision an SSL cert until you are hosting from your domain.

Solution

The easiest solution is to initially generate your SSL cert manually. You can then use something like certbot-s3front to auto-renew.

Details

Continue reading "Bootstrapping S3/CloudFront with LetsEncrypt"

Linux PowerShell on Linux doesn't find modules when run through Puppet

Posted by Admin • Thursday, March 11. 2021 • Category: DevOps, Linux

I'm running PowerShell on Linux for the sake of using PowerCLI. An interesting thing happens: when pwsh is run interactively, in a terminal, the following works fine when PowerCLI is already installed:

pwsh -c 'Get-InstalledModule VMware.PowerCLI'

When puppet runs it - it doesn't find the module:

Get-Package: /opt/microsoft/powershell/7/Modules/PowerShellGet/PSModule.psm1:9445
Line |
9445 |          PackageManagement\Get-Package @PSBoundParameters | Microsoft. …
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | No match was found for the specified search criteria and
     | module names 'VMware.PowerCLI'.

Error: 'pwsh -c "if (-not(Get-InstalledModule VMware.PowerCLI)) { Exit 1 }"' returned 1 instead of one of [0]

After some experimentation, I narrowed it down to a single environment variable missing when puppet executes the command: HOME. Set HOME, and powershell will find modules. That gets us the following recipe:

  $my_user = 'some_user'

  exec { 'Install PowerCLI':
    path    => ['/bin', '/usr/bin'],
    command => 'pwsh -c \'Install-Module -Force:$true -Name VMware.PowerCLI\'',
    unless  => 'pwsh -c \'if (-not(Get-InstalledModule VMware.PowerCLI -ErrorAction silentlycontinue)) { Exit 1 }\'',
    user    => $my_user,
    environment => [ "HOME=/home/${my_user}"]
  }

Java Gradle Toolchains Support - different JVMs for compile and test

Posted by Admin • Thursday, February 4. 2021 • Category: DevOps, Java

I'm testing a product that needs to be compiled with JDK 8 but tested (sometimes) on JDK 11. This is now possible to do with maven surefire (although that took some effort). With gradle, I was doing it as follows, which is terrible, even if the path comes from configuration:


// The old way:
test {
    executable = '/some/hardcoded/path/to/java'
}
 

Now that Gradle 6.7+ has built-in toolchains support, it's trivial to configure the compile toolchain:


java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(11)
    }
}
 

But what about tests? What if I want to test with a different JVM? It's a little more verbose:


test {
    JavaToolchainService javaToolchainService = project.getExtensions().getByType(JavaToolchainService.class)
    def launcher = javaToolchainService.launcherFor{
        languageVersion = JavaLanguageVersion.of(11)
    }
    javaLauncher = launcher
    environment 'JAVA_HOME', launcher.get().metadata.installationPath //if your tests care about JAVA_HOME
}
 

Linux Puppet recipe for setting up autossh via systemd

Posted by Admin • Thursday, November 19. 2020 • Category: DevOps, Linux

I've always set up autossh in /etc/rc.local, but with CentOS 8 that doesn't work well (things start too early, etc). Luckily, there is a nicer way using systemd templates. Essentially, all you have to do is create one symlink and one config file per instance of autossh.

Example:

Make a config file


/etc/autossh/mything.conf:

OPTIONS=-N  -M 20000 -R8888:1.2.3.4:8888 5.6.7.8


Make a symlink

ln -s /usr/lib/systemd/system/[email protected] /etc/systemd/system/[email protected]


Test

systemctl start [email protected]
journalctl -xe


But it's better to automate:

Puppet Recipe



define autossh(
    $service=$name,
    $args,
  ) {

    file {"/etc/systemd/system/autossh@${service}.service":
      ensure => link,
      target => '/usr/lib/systemd/system/[email protected]'
    } ->
    file {"/etc/autossh/${service}.conf":
      content => "OPTIONS=${args}"
    } ->
    service {"autossh@${service}":
      ensure => running,
      enable => true,
    }

}

# and example usage - random port forwarding
autossh{'mything': args => '-N  -M 20000 -R8888:1.2.3.4:8888 host1'}
autossh{'mything2': args => '-N  -M 20000 -R8888:1.2.3.4:8888 host2'}
 

Using puppet to set Windows Computer Description

Posted by Admin • Monday, May 18. 2020 • Category: DevOps

My company's security department decided to assign Antivirus exclusion policies based on the value of the windows computer description. That means that I need to set computer descriptions ( net config server /srvcomment:"new description" ) to the same value on a whole bunch of windows machines. Doing that by hand is unappealing, and I already have puppet, so here is a simple solution:


MYCLASS/lib/facter/wincomputer_description.rb


# Gets computer decscription from 'net config server'
Facter.add("win_computer_description") do
  setcode do
    output = `net config server`
    output.split(/\n/).find{|it| it.start_with?('Server Comment') }.gsub(/Server Comment[\s]*/,'')
  end
end
 

MY_CLASS/manifests/init.pp


class MY_CLASS(
  $computer_description = 'Some-new-description',
) {
  if ($win_computer_description != $computer_description) {
    exec { 'Set computer description':
      path => ["C:\\windows\\system32"],
      command => "net config server /SRVCOMMENT:\"${computer_description}\"",
    }
  }
}
 

Disabling Windows Recycle Bin with Puppet on all versions of Windows

Posted by Admin • Thursday, March 22. 2018 • Category: DevOps

And when I say "All versions of Windows" I mean that I tested it on Server 2008, 2012 and 2016.



This was oddly hard to figure out, and most tutorials either apply to only one version of windows or to outdated tools. The best way to do this that I found was using Local Group Policy. Now, how to automate this? The puppet localgrouppolicy module didn't work at all when I tried it (and has not been updated since 2014). The proper way to do this is of course with a Domain-based Group Policy, but my machines are not members of a domain.



Fortunately, there is a new Microsoft tool called LGPO that allows for some degree of command-line control of the Local Group Policy. Download "LGPO.exe" here.

First, let's make a reusable policy text file that we can import on all machines:

  1. take a vanilla Windows machine that hasn't had any Group Policy customization, and use lgpo.exe to export the policy: "lgpo.exe /v /parse /u c:\windows\system32\GroupPolicy\User\Registry.pol" (at least that was appropriate in my case). You should get more or less empty output.
  2. Then use the Local Group Policy Editor to change "Do not move deleted files to the Recycle bin" (under User Configuration -> Administrative Tools -> All Settings) to "Enabled"
  3. Repeat step 1. You should see this one setting that you changed in the output. Redirect output to a file, this will be our text file
  4. You can test that importing this file will change the setting: "lgpo /r myfile.txt". (Change the setting back first, run this, then re-open the Local Group Policy Editor to see the change)

Now, we can set up puppet:

Continue reading "Disabling Windows Recycle Bin with Puppet on all versions of Windows"

Jenkins Pipeline: parallel and waitUntil, waiting until the other branch finishes

Posted by Admin • Monday, January 22. 2018 • Category: DevOps

Let's say that for the sake of speed, you are running two slow things in parallel, but you want one to wait for the other.


parallel one: {
    node {
        sh "sleep 15"
    }
}, two: {
    node {
        //slow part:
        sh "sleep 10"  

        // now do something that needs "one" to finish.  There is a good chance that this will run too soon...
        sh 'wget http://something' //for example
    }
}
 

The problem, obviously, is that you can't be sure that the wget will run after part one finishes. (Let's assume that part one creates the file).

Continue reading "Jenkins Pipeline: parallel and waitUntil, waiting until the other branch finishes"

Java Linux Running Jenkins Swarm client as a service via Upstart

Posted by Admin • Wednesday, December 21. 2016 • Category: DevOps, Java, Linux

This turned out to be fairly simple, with only one gotcha: do not follow the how-to's out there that tell you to use expect fork. The process doesn't technically fork. When I had that setting enabled, upstart commands would hang under very specific but repeatable conditions (if the process was killed externally).



So, here is my upstart conf file:

Continue reading "Running Jenkins Swarm client as a service via Upstart"

Linux Docker: Automatically remove containers that have been running too long

Posted by Admin • Thursday, October 20. 2016 • Category: DevOps, Linux

Why Because my Jenkins setup sometimes starts containers and forgets about them. Either it thinks it failed to start one, or the container itself has trouble starting. Either way, I'm left with containers that are running, trying to connect to Jenkins in vain, forever. The proper way to fix this is probably to have the containers timeout at some point, but that mechanism is broken.



Anyway, the fix I have is a true hack: find containers that have been up more than 2 days and kill them. None of our jobs should run for more than about a day, so this is a safe limit. Here is a bash script to do this:

Continue reading "Docker: Automatically remove containers that have been running too long"

Linux Jenkins command line ssh: Host key verification failed despite ssh-agent

Posted by Admin • Thursday, August 4. 2016 • Category: DevOps, Linux

After hours of "Why does it work locally but not in Jenkins", this error boils down to StrictHostKeyChecking... In other words, since the job runs as a user on a random slave, and this user most likely doesn't have a known hosts file with an entry for the target system, this fails rather cryptically. You think that the user's keys don't work, but that's not the problem.



The whole setup boils down to :

  1. Install ssh-agent plugin
  2. Configure credentials with a valid ssh key for your target
  3. Enable ssh-agent with that credential entry in your job config
  4. Add StrictHostKeyChecking=no to whatever ssh command you are using. Some examples:
    • GIT:
      export GIT_SSH_COMMAND="ssh -oStrictHostKeyChecking=no"
    • SSH:
      ssh -oStrictHostKeyChecking=no ....

Code and Hacks Marking Jenkins build UNSTABLE from environment inject groovy script

Posted by Admin • Wednesday, March 23. 2016 • Category: Code and Hacks, DevOps
The way we use the Jenkins "Prepare an environment for the run" feature of the Environment Inject Plugin is this: we use the "Evaluated Groovy Script" to retrieve some info from a successful run of some other job (svn revisions, environment vars, whatever) in order to make it available to the environment of the current build. But what if that other job hasn't run successfully (yet)? I'd like to distinguish this "soft failure" from a real problem, so I'd like the build status to be "UNSTABLE". Here is how to do that in env inject:
currentBuild.@result = hudson.model.Result.UNSTABLE
However, that will not terminate the build. If you want to stop execution at this point, you'll have to put conditional logic in the job steps (but you can set a variable in EnvInject, and, for example, check it using Conditional Step)

Alternatively, to mark the build aborted and terminate immediately instead:

Screenshot:

Continue reading "Marking Jenkins build UNSTABLE from environment inject groovy script"

Linux Migrating Maven Jenkins jobs to FreeStyleJobs due to JDK 1.6 incompatibility

Posted by Admin • Tuesday, March 15. 2016 • Category: DevOps, Linux
Soon after Jenkins 1.609.1 support for JDK 1.6 was dropped altogether. This means that you can run jobs with whatever JDK you want as long as you are not using the Maven job type. Maven jobs require JDK 1.7 or higher (actually they run on whatever version the master is using, ignoring the JDK setting in the job configuration). This is a big deal for a shop that does extensive cross-platform testing. There are two solutions:
  1. Convert hundreds (in our case) of Maven jobs to regular (freestyle) jobs. There appears to be no simple script to do this, so I wrote one here
  2. Use maven toolchains to compile and test with a different JDK. Although converting jobs is a viable option that requires no additional setup, using toolchains may be preferred in some cases. For one thing, Jenkins offers the "Perform Release" button on maven jobs, something that is difficult to emulate in freestyle jobs.
Regarding option 1:

Initially I tried doing this using JobDSL, but that is way too time-consuming and hard to maintain. The simplest approach is to take the job XML, make minimal changes and push it back. Except that it's not that simple - you can't change the job type. You need to create new jobs, which is why I wrote a script to do all of the above. It will move the pre/post builders to the main builder list, and create a new maven job step in between. The rest is left unchanged and most of the XML is naturally valid for this version of jenkins/plugins since it's pulled from the live Jenkins.

Java Linux Jenkins: Bulk editing jobs to remove a trigger

Posted by Admin • Thursday, December 17. 2015 • Category: DevOps, Java, Linux
I have about 200 jobs that have the HudsonStartupTrigger (this is a plugin) turned on. This makes all of them run every time the master is restarted, causing the build queue to go crazy. I don't know why people turned this on in so many jobs (probably blindly copying jobs) over the years, but I'd like it gone. I don't want to restart the master or uninstall the plugin.

Here is a script console snippet to do this (can be adapted to remove other triggers easily). This does not do folders, if you're using that plugin.


import hudson.triggers.*
 
for (item in Hudson.instance.items) {
    name = item.name
       
  if (item instanceof AbstractProject) {
    triggers = ((AbstractProject)item).getTriggers()
   
    triggers.each{descriptor, trigger ->
      if (descriptor instanceof org.jvnet.hudson.plugins.triggers.startup.HudsonStartupTrigger$HudsonStartupDescriptor) {
            out.println("Removing startup trigger from job " + name)
        ((AbstractProject)item).removeTrigger(descriptor)
      }
    }
  }
}
 

Linux Suse11 cannot mount NFS share

Posted by Admin • Wednesday, December 9. 2015 • Category: DevOps, Linux
This is a recent issue on Suse 11.3. I have hundreds of machines mounting the same share fine - Centos, RedHat, even AIX.
mount -v /mountpoint                                                                                                               
mount.nfs: timeout set for Wed Dec  9 11:09:27 2015
mount.nfs: trying text-based options 'rsize=8192,wsize=8192,intr,hard,addr=X.X.X.X'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=2, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: requested NFS version or transport protocol is not supported


Suprisingly, the problem is exactly what it says it is. Adding ",vers=4" to the options takes care of the problem. Hope this saves you time.