Challenges

Scientific evidence (expert testimony) is dealt with in FRE rules 701-706 and the Frye and Daubert cases as well as elsewhere.[3][4][5]

In order to be admitted, digital forensic evidence must survive challenges to relevance, authenticity, its hearsay nature, the original writing requirement, must not be far more prejudicial than it is probative, and must be introduced and analyzed by people who meet standards. It is incumbent on the party introducing evidence to meet these criteria and on the party challenging to oppose based on these criteria and to do so in a timely fashion as part of the legal process. Experts can help make this happen by identifying all lines of challenge and providing expert analysis, advice, knowledge, and skills to help create the conditions for challenges.

In cases where there is a lot at stake for the parties involved, DFE is likely to be challenged in significant ways. The basic challenges to DFE can be made to a greater or lesser extent at every step of the process, for every item of evidence, and for every witness presented. The challenges may be thought of in terms of a specific set of known fault types that form a fault model. [1]

Make / Miss: In the fault model discussed in [1] faults are characterized as errors of omission, commission, or combinations thereof, sometimes called errors of substitution. Errors of omission are also called "miss" faults because they miss an evidence identification, collection, preservation, transportation, storage, analysis, interpretation, attribution, reconstruction, presentation, or destruction (process) step or miss content, context, meaning, relationship, ordering, time, location, corroboration, or consistency results. Errors of commission are also called "make" faults because they introduce evidence process steps that should not be present or assert content, context, meaning, relationship, ordering, time, location, corroboration, or consistency results that are not real.

Content: Making content typically involves processing errors. For example, uncleaned media is used in the analysis process and the analysis finds evidence that was left over from a previous case. This was addressed under imaging, above. The challenge to this can come in many forms, and if original evidence or cryptographic checksums are not used, such challenges have a good chance of success because of the inability to independently verify results. If originals are present and checksums can be shown to match, then such challenges will likely only succeed in the presence of an actual and material analysis error, because the purity of the evidence can be usually properly established. [1]

Missing content typically results from limited time or excessive focus of attention. Limited time is almost always an issue because there is usually an enormous amount of evidence present, most of which can only be peripherally examined with simple tools. Examining every bit pattern from every possible perspective is simply too time consuming to be feasible and is almost never necessary to get to the heart of the evidence. Excessive focus, on the other hand, is easier to avoid. By simply taking an open view of what could be meaningful evidence and being thorough in the evaluation process, such misses are avoidable. The challenge is simple. Did you look at everything? Is there any exculpatory evidence? Where did you look? Why did you not look in the other places? What technique did you use? Why did you not use a more definitive technique? Is there a more definitive technique? The questions can be nearly endless. [1]

Context: Information only has meaning in context. Analysis can make context by making assumptions that are invalid or cannot be demonstrated. Context is missed when assumptions that are valid and can be demonstrated are not made. The challenge to made context starts with questioning the basis for assumptions. If assumptions cannot be adequately demonstrated, the context becomes dubious, the assumptions fall away, and the conclusions are not demonstrable. If an alternative context can be demonstrated with the same or better basis, that context can be substituted and the interpretation of the evidence altered. Missed context can be challenged with the introduction of alternative contexts. It then becomes the challenge of the other side to disprove these contexts. [1]

Meaning: The meaning of things that are found is obviously the basis for interpretation. Meaning that is missed leads to a failure to interpret, and meaning that is made is an interpretation without adequate support. [1]

Process: Content does not come to exist through magic. It comes to exist through a process. The notion that a sequence of bits appears on a system without the notion of how that sequence came to exist there makes for a very weak case. If the bits were created within the system, the means for their creation should be there unless it was somehow removed. If the bits were obtained from somewhere else, the process by which they got there should be identified. If there are alternative explanations for the arrival of the bit sequence, why is one interpretation better than the other? [1]

Relationship: Just as sequences of events produce content, relationships between event sequences and content produce content. The presence or absence of related content causes differences in the content generated by related processes. The presence or absence of a directory prior to running a program that uses or creates it produces a difference in the time associated with the creation of the directory. Similarly, the placement of the directory in the linked lists associated with the file system relative to the placement of files within that directory may indicate the differences in these relationships. There are many such relationships within systems and those relationships can be explored to challenge the assertions of those who make claims about them. [1]

Ordering: Sequences are a special case of orderings. More generally, orderings can involve things that cannot be differentiated from being simultaneous, while sequences are completely ordered. Timing often cannot be established with perfection, but partial orderings can be derived. The possible orders of events can make an enormous difference in some cases. One obvious reason for this is that ordering is a precondition for cause and effect. To assert that one thing caused another, it must be demonstrable that the cause preceded the effect. If this cannot be established by timing there is the potential to challenge based on the lack of a causality. [1]

Time: The most common challenge to computer-related times stems from the potential difference between a computer clock and the real- world time. Even accounting for time zone variations this is an all too common problem that has to be addressed in the forensic process. If the time reference for the computer is not established at the time the evidence is collected, timing can sometimes be obtained by relating the timing of events within the computer to externally timed network events. Missed time can sometimes be made up for by correlation with outside events, while made time can often be demonstrated wrong by similar correlation. The lack of correlating information represents sloppiness in the collection and analysis process that may itself lead to the inability to determine timing. [1]

Location: Everything that happens in computers has physicality despite any efforts to portray it as somehow ephemeral. Physicality tends to leave forensic evidence in one form or another. For example, when a person uses a keyboard, particles from hair and skin fall into the keyboard and tend to get stuck there. In a similar fashion, data in computers tends to be placed on the disk and tends to get stuck there. Computer systems have physical characteristics as well, and sometimes they are dead giveaways to location.

Corroboration: There is a great deal of corroborating evidence that can be sought from connected systems that produce log files that can confirm or refute the use of a system by a suspect. If the evidence is not sought and the actions are in question, either in terms of taking place or in terms of their source, path, or content, the lack of intermediate audit trails may complicate the ability to definitively show what took place.

Consistency: If a program is asserted to generate a file that was not otherwise altered, then the program must have been running at the time the file was created, must have had the necessary permissions to create the file, must have the capacity to create such a file in such a format, and must have been invoked by a user or the system using another program capable of invoking it. There is a lot of information that should all link together cleanly. If it doesn't, there are reasons to question it. [1]

This is not to say that all of these records always exist in the proper order on all systems. For various reasons, some records get lost, others end up out of order, and times fluctuate to some extent; however, these are all within reasonable expected tolerances and substantial deviations are often detectable. Such deviations are indicators that things are not what they seem, and in such cases alternative explanations are available and should be pursued. [1]

Accident / Intent: Accidental miss faults are practically impossible to avoid because there are a potentially unlimited number of different analytical methods and processes that could be applied to evidence, any of which might produce something of relevance.

Accidental make faults are normally the result of inadequate attention to detail, lack of expertise, a non-systematic process, or a lack of thoroughness. These faults are particularly problematic because they produce interpretations that claim things that are not true. The lack of adequate time to thoroughly investigate issues leads to make faults because, in the process of investigation and analysis, theories are produced and tested. The human mind tends to make leaps that are the source of human intelligence, but these leaps may or may not be right. A lack of time, care, or expertise, leads to the acceptance of these theories as if they were facts without adequate verification, or their presentation as definitive when they remain somewhat speculative.

Intentional miss faults are commonplace, particularly in adversarial situations. Each side tends to leave out the things that the other side might find helpful to their case and to focus on the issues that best make their own case. Counsel sometimes limits the information available to DFE experts so that they only see the things that tend to aid the client in their case. The DFE expert should be aware that limited information leads to excessive conclusions and take care in drawing conclusions to explicitly state the limits of their conclusions and their basis. If the basis changes, so might the conclusions. Experts who intentionally ignore facts in front of them and draw conclusions that are contradicted by those facts are likely to face serious and justified challenges.

Intentional make faults are almost always fraudulent in nature. Making up evidence or creating conclusions that the expert knows to be false are unethical and in most cases illegal and sanctionable. The DFE expert should seek to identify intentional make faults by verifying results using redundant methods and verifying evidence consistency through analytical methods. Intentional miss faults are often used to cover up intentional make faults. For example, when identifying evidence, such as log files associated with computers that generated other evidence in the case, the party who produces detailed records of one sort but refuses to provide, intentionally destroys, or fails to adequately retain records of related sorts, should be suspected of fabricating the detailed evidence that they proffer. The DFE expert should identify this issue clearly and assert the potential of spoliation of the detailed evidence provided. If that evidence has internal inconsistencies, the case for intentional spoliation becomes stronger.

False Positives: False positives are results indicating something as true when in fact it is not true. For example, the detection of a condition when the condition was never in fact present, the attribution of an action to a party who did not in fact take that action, or the claim of the presence of contraband when in fact it was not present.

False Negatives: False negatives are results indicating that something was not true when in fact it was true. For example, the failure to detect the presence of a break-in to a computer that was supposed to be reliably storing evidence when claiming that the computer was not broken into, the failure to attribute an action to an actor when it can in fact be attributed reliably based on available information, or the claim of absence of contraband when contraband is in fact present.

In many cases, these sorts of errors are the result of DFE experts making statements that are overly broad, excessively definitive, or otherwise stated as unilateral and sweeping when they are in fact accurate only for a more limited set of conditions. But in other cases, these are simply the result of process errors in which some key piece of evidence was not properly identified, collected, preserved, etc. or in which something that was not in fact reliable was treated as if it were reliable.

Faults are important to legal matters when they produce erroneous results or conclusions. The mere presence of an accidental miss does not imply that the expert drew incorrect conclusions or that the evidence doesn't support the matter at hand. In order for a fault to rise to the level of importance that makes it worthy of a legal challenge, that fault should normally produce an error that is material to the case. Even intentional fabrication of evidence doesn't always produce errors that are material. For example, someone who accidentally destroyed a file and created a new version in its place without telling anyone, augmented their accidental miss into an intentional make, but that doesn't mean that the result was inaccurate, only that its pedigree is questionable.

The DFE expert should identify relevant faults, but it is far more important to identify the faults that produce errors and put those errors into the proper legal context. The net effect of faults that are meaningful can be characterized in terms of two kinds of errors; false positives and false negatives.