[Servercert-wg] Ballot SC-75 - Pre-sign linting

Mon Jun 3 11:22:46 UTC 2024

On 3/6/2024 12:25 μ.μ., Rob Stradling wrote:
> > Do people have strong feelings against RECOMMENDing linting of 
> currently-valid certificates when linting software gets updated, with 
> a threshold of 90 days after the update of the linting software?
>
> ISTM that some people tend to read "SHOULD" or "RECOMMENDED" as "we 
> strongly urge you to do this thing, but don't worry if you don't 
> because it's 100% optional".  IMHO, this interpretation is nearer to 
> "MAY" than to the real definition of "SHOULD".
>
> "SHOULD" needs to be understood as "100% required, except for 'valid 
> reasons in particular circumstances'" (quoting RFC2119), and I think 
> I'm right in saying that in CABForum documents we often use "SHOULD" 
> to define requirements that we anticipate will become "MUST" 
> requirements in the future.
>
> With that in mind, I do not support RECOMMENDing something that is 
> known to be technologically infeasible for high-volume certificate 
> issuers.

Thanks for the feedback Rob. These "high-volume certificate issuers" ARE 
within the "particular circumstances" and IMHO within the spirit and 
letter of RFC 2119. When I first recommended a SHOULD, I interpreted 
this to be 100% required, except for valid reasons in particular 
circumstances. In any case, I will not insist on a SHOULD if ppl have 
strong objections.

>
> > CAs with such large volumes could perform sampling and pick a 
> smaller number of certificates for linting during the linter update
>
> Imposing a minimum percentage sample size (even as low as 1%) might be 
> challenging for some high-volume CAs.  Perhaps we could solve that by 
> letting CAs pick their own sample sizes, but TBH I'm not convinced 
> that sampling is the best or only approach.

This is a pretty standard audit practice when there are technical or 
other practical barriers in auditing large numbers of 
artifacts/populations. CAs (including high-volume CAs) are already 
required to perform a 3% sampling of issued certificates per quarter. 
The expectation of the requirement is to highlight that the CA should 
make sure they have not misissued certificates due to their at-the-time 
linter having bugs or not including specific lints that the update now 
supports. It could be left for the CA to decide whether to lint the 100% 
of valid certificates or a 3% (or zero, if they can justify the exceptions).

>
> ISTM that the goals here are (1) to avoid rolling out buggy linters to 
> production CA systems and (2) to assist linter developers with 
> discovering and fixing bugs in linters.  So why don't we encourage CAs 
> to look closely at the changelog of each new linter version and then 
> do targeted testing of the lints that have been added or changed?  
> i.e., white box testing instead of black box testing.  Perhaps we 
> could even encourage CAs to review the changes to the linter's source 
> code too.  I think these sorts of approaches are more likely to 
> uncover linter bugs, and more quickly.
>
> But...
> Whilst it's great to encourage the CA community to collectively engage 
> in helping to improve the quality of linters, I'm not convinced it's 
> appropriate to define any sort of "SHOULD" requirement in the BRs 
> relating to this.  Similarly, whilst it's great to encourage the CA 
> community to collectively engage in supporting the CT ecosystem, we 
> don't have a "The CA SHOULD operate a CT log" requirement specified 
> anywhere.

I also agree that the BRs are not the right place to enforce 
requirements/expectations for CAs to contribute collectively in linting 
or other tools.

Dimitris.

>
> ------------------------------------------------------------------------
> *From:* Servercert-wg <servercert-wg-bounces at cabforum.org> on behalf 
> of Dimitris Zacharopoulos (HARICA) via Servercert-wg 
> <servercert-wg at cabforum.org>
> *Sent:* 03 June 2024 06:13
> *To:* Aaron Gable <aaron at letsencrypt.org>; CA/B Forum Server 
> Certificate WG Public Discussion List <servercert-wg at cabforum.org>
> *Subject:* Re: [Servercert-wg] Ballot SC-75 - Pre-sign linting
> CAUTION: This email originated from outside of the organization. Do 
> not click links or open attachments unless you recognize the sender 
> and know the content is safe.
>
>
>
> On 1/6/2024 12:20 π.μ., Aaron Gable wrote:
>
>     On Wed, May 29, 2024 at 10:57 PM Dimitris Zacharopoulos (HARICA)
>     via Servercert-wg <servercert-wg at cabforum.org
>     <mailto:servercert-wg at cabforum.org>> wrote:
>
>         While we're in this vein, it would also be useful to add a
>         recommendation for CAs to lint all non-expired, non-revoked
>         certificates whenever they install an update of their linting
>         software.
>
>           * "The CA SHOULD perform Linting on the corpus of its
>             non-expired, non-revoked Subscriber Certificates whenever
>             it updates the Linting software".
>
>         What do people think about these proposals?
>
>
>     Just chiming in to say that I don't love this proposal, for a few
>     reasons.
>
>     1. Linting software has not always had a great track record of
>     applying new lints (based on new requirements) only to
>     certificates issued after a certain date. Running a new linting
>     tool over old certificates frequently raises warnings or errors
>     which were not actually errors at the time of issuance. Zlint has
>     support for this behavior, but it is not used consistently across
>     all lints in their corpus. A quick glance at pkilint's source does
>     not seem to show any support for this behavior, but I easily could
>     be wrong.
>
>
> Linters can be expected to throw false positives and it's the CA's 
> responsibility to interpret those results properly. Linting the 
> non-expired, non-revoked certificates is usually a "one-off" task 
> (linting software is usually not updated so frequently) and if the CA 
> decides that a certain lint is a false positive, it could be excluded 
> for that run. In most cases though, updated linters may catch past 
> compliance matters and mis-issuances that are correctly detected and 
> reported. It's always best to risk some false positives (which you can 
> document and move on) than to risk missing real misissuances which 
> will ultimately be revealed by third parties scanning the CT logs.
>
>
>     2. Some CAs have very large certificate corpuses, e.g. Let's
>     Encrypt has 400 million currently-valid certificates. Some linting
>     tools are very slow, e.g. pkilint's `lint_pkix_cert` takes 300ms
>     per run. At that rate, re-linting LE's whole corpus would take
>     /four years/. I'm sure there are speedups to be had, but they'd
>     have to be several orders of magnitude to make that feasible.
>
>
> I believe this is a valid case for CAs that have large certificate 
> volumes and linting every single currently-valid certificate is not a 
> viable option, which is why this is a SHOULD requirement. Even so, 
> based on audit methodologies, LE and other CAs with such large volumes 
> could perform sampling and pick a smaller number of certificates for 
> linting during the linter update.
>
>
>     3. Any large systems engineer knows that streaming processing and
>     batch processing infrastructure are very different, with wholly
>     different software and hardware setups to make each efficient. I
>     think it is much more important to incentivize stream-linting
>     (i.e. as issuance happens), and that it would be counterproductive
>     to require CAs to invest in both at the same time.
>
>
> I'm probably thinking about this a bit differently because linting can 
> be executed in multiple ways. It's the same software that can be 
> executed in the issuance pipeline (pre-sign linting) and after the 
> issuance pipeline (post-sign linting). In the latter case, it can be 
> executed in batch mode that checks multiple certificates. The ballot 
> certainly puts more weight on the stream-linting process but also 
> recommends post-sign linting, at least when the linting software gets 
> updated. I believe this is a good balance that puts all the 
> expectations of using linting tools in the BRs.
>
> Do people have strong feelings against RECOMMENDing linting of 
> currently-valid certificates when linting software gets updated, with 
> a threshold of 90 days after the update of the linting software?
>
>
> Dimitris.
>
>
>
>     Thanks,
>     Aaron
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cabforum.org/pipermail/servercert-wg/attachments/20240603/2cd973c4/attachment.html>